Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14035

Enable predicate pushdown to delta files created by ACID Transactions

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • Transactions

    Description

      In current Hive version, delta files created by ACID transactions do not allow predicate pushdown if they contain any update/delete events. This is done to preserve correctness when following a multi-version approach during event collapsing, where an update event overwrites an existing insert event.
      This JIRA proposes to split an update event into a combination of a delete event followed by a new insert event, that can enable predicate push down to all delta files without breaking correctness. To support backward compatibility for this feature, this JIRA also proposes to add some sort of versioning to ACID that can allow different versions of ACID transactions to co-exist together.

      Attachments

        1. Design.Document.Improving ACID performance in Hive.02.docx
          24 kB
          Saket Saurabh
        2. Design.Document.Improving ACID performance in Hive.01.docx
          24 kB
          Saket Saurabh
        3. HIVE-14035.17.patch
          197 kB
          Saket Saurabh
        4. HIVE-14035.16.patch
          197 kB
          Saket Saurabh
        5. HIVE-14035.15.patch
          197 kB
          Saket Saurabh
        6. HIVE-14035.14.patch
          197 kB
          Saket Saurabh
        7. HIVE-14035.13.patch
          166 kB
          Saket Saurabh
        8. HIVE-14035.12.patch
          158 kB
          Saket Saurabh
        9. HIVE-14035.11.patch
          152 kB
          Saket Saurabh
        10. HIVE-14035.10.patch
          115 kB
          Saket Saurabh
        11. HIVE-14035.09.patch
          116 kB
          Saket Saurabh
        12. HIVE-14035.08.patch
          117 kB
          Saket Saurabh
        13. HIVE-14035.07.patch
          87 kB
          Saket Saurabh
        14. HIVE-14035.06.patch
          105 kB
          Saket Saurabh
        15. HIVE-14035.05.patch
          94 kB
          Saket Saurabh
        16. HIVE-14035.04.patch
          90 kB
          Saket Saurabh
        17. HIVE-14035.03.patch
          119 kB
          Saket Saurabh
        18. HIVE-14035.02.patch
          94 kB
          Saket Saurabh
        19. HIVE-14035.patch
          91 kB
          Saket Saurabh

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HIVE-14878 integrate MM tables into ACID: add separate ACID type

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14448 Queries with predicate fail when ETL split strategy is chosen for ACID tables

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-16832 duplicate ROW__ID possible in multi insert into transactional table

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. SPARK-15348 Hive ACID

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-17458 VectorizedOrcAcidRowBatchReader doesn't handle 'original' files

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is required by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14667 Eliminate OrcRecordUpdate.OPERATION

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14607 ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 1

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12631 LLAP IO: support ORC ACID tables

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HIVE-14516 OrcInputFormat.SplitGenerator.callInternal() can be optimized

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HIVE-14233 Improve vectorization for ACID by eliminating row-by-row stitching

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-17204 support un-bucketed tables in acid

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-17089 make acid 2.0 the default

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14199 Enable Bucket Pruning for ACID tables

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-15988 LLAP: cache ORC ACID delete deltas

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14534 modify tables in tests in HIVE-14479 to use transactional_properties=default

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-16669 Fine tune Compaction to take advantage of Acid 2.0

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HIVE-16830 delete delta buckets don't have maxLength set

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-14514 OrcRecordUpdater should clone writerOptions when creating delete event writers

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link ReviewBoard # 49766

          Activity

            People

              saketj Saket Saurabh
              saketj Saket Saurabh
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: