Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15789

Vectorization: limit reduce vectorization to 32Mb chunks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • Vectorization
    • None
    • Vectorization: limit reduce vectorization to 32Mb chunks (Teddy Choi, via Gopal V)

    Description

      Reduce vectorization accumulates 1024 rows before forwarding it into the reduce processor.

      Add a safety limit for 32Mb of writables, so that shorter sequences can be forwarded into the operator trees.

              rowIdx++;
        if (rowIdx >= BATCH_SIZE) {
          VectorizedBatchUtil.setBatchSize(batch, rowIdx);
          reducer.process(batch, tag);

      Attachments

        1. HIVE-15789.2.patch
          4 kB
          Teddy Choi
        2. HIVE-15789.1.patch
          4 kB
          Teddy Choi

        Issue Links

          Dependent

          Bug - A problem which impairs or prevents the functions of the product. HIVE-15700 BytesColumnVector can get stuck trying to resize byte buffer

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review Board

          Activity

            People

              teddy.choi Teddy Choi
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: