Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.2.0, 4.0.0
-
None
Description
Where there are too many columns (in the order of 100s), with decimal, string types the column vector pool of buffers created here https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59 can cause memory pressure.
Example:
128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB
The pool size keeps increasing when there is slow consumer but fast llap io (SSDs) leading to GC pressure when all LLAP io threads read splits from same table.
Attachments
Attachments
Issue Links
- links to