Understand usage of netty's onheap memory use, even with offheap pools

We've observed that netty uses large amount of onheap memory in its pools, in addition to the expected offheap memory when I added some instrumentation (using SPARK-24918 and https://github.com/squito/spark-memory). We should figure out why its using that memory, and whether its really necessary.

It might be just this one line:
https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java#L82

which means that even with a small burst of messages, each arena will grow by 16MB which could lead to a 128 MB spike of an almost entirely unused pool. Switching to requesting a buffer from the default pool would probably fix this.