Rate of JMS Messages consumed by TomEE drops significantly over time

Hey guys,

We're noticing a pretty strange issue processing a large number of JMS
messages. After about 20k messages, messages consumed per second drops off
and there's heavy GC activity (smells like a memory leak). What interesting
though is the server continues to run and doesn't OutOfMemoryError. A simple
restart of TomEE (but not the broker) temporarily fixes the problem. We're
using an external broker, not the internal one

What's interesting, is that Jonathan Gallimore and I were talking about a
similar issue with Websockets. We noticed that eventually the servers will
exhibit the same behavior once enough websockets are opened and closed. This
issue occurs infrequently enough because we might handle 20,000 websockets
over the course of a few days, but we can process 20,000 JMS messages in a
few mins.

I have a heap dump from the issue and several jstacks. I'll be honest, I'm
not sure where to start. In the past I've solved memory leaks by careful
code audits. Our codebase happens to be mostly stateless, with everything
else being managed by CDI scopes (ApplicationScoped and TransactionScoped).

What's a good way to get started?

http://tomee-openejb.979440.n4.nabble.com/Performance-issue-with-JMS-on-Tomee-7-0-5-td4687296.html