Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21391

LLAP: Pool of column vector buffers can cause memory pressure

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 4.0.0
    • None
    • llap

    Description

      Where there are too many columns (in the order of 100s), with decimal, string types the column vector pool of buffers created here https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59 can cause memory pressure. 

      Example:

      128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB

      The pool size keeps increasing when there is slow consumer but fast llap io (SSDs) leading to GC pressure when all LLAP io threads read splits from same table. 

      Attachments

        1. HIVE-21391.1.patch
          5 kB
          Prasanth Jayachandran
        2. HIVE-21391.2.patch
          26 kB
          Slim Bouguerra

        Issue Links

          Activity

            People

              bslim Slim Bouguerra
              prasanth_j Prasanth Jayachandran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h