Description
Discussion from Github issue: https://github.com/apache/orc/issues/1240
In short, CompressionStream has fixed its input buffer to compression block size (which is 1M by default). Check here for reference: https://github.com/apache/orc/blob/main/c%2B%2B/src/Compression.cc#L108 .
The piece of code can be improved by replacing its type from DataBuffer by BlockBuffer. In this way, it is able to start with a small initial size and automatically grow as needed.
Attachments
Issue Links
- links to