Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20699

Query based compactor for full CRUD Acid tables

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • None
    • Transactions
    • None
    • Reviewed

    Description

      Currently the Acid compactor is implemented as generated MR job (CompactorMR.java).

      It could also be expressed as a Hive query that reads from a given partition and writes data back to the same partition. This will merge the deltas and 'apply' the delete events. The simplest would be to just use Insert Overwrite but that will change all ROW__IDs which we don't want.

      Need to implement this in a way that preserves ROW__IDs and creates a new base_x directory to handle Major compaction.

      Minor compaction will be investigated separately.

      Attachments

        1. HIVE-20699.1.patch
          23 kB
          Vaibhav Gumashta
        2. HIVE-20699.1.patch
          25 kB
          Vaibhav Gumashta
        3. HIVE-20699.2.patch
          41 kB
          Vaibhav Gumashta
        4. HIVE-20699.3.patch
          58 kB
          Vaibhav Gumashta
        5. HIVE-20699.4.patch
          58 kB
          Vaibhav Gumashta
        6. HIVE-20699.5.patch
          59 kB
          Vaibhav Gumashta
        7. HIVE-20699.6.patch
          59 kB
          Vaibhav Gumashta
        8. HIVE-20699.7.patch
          59 kB
          Vaibhav Gumashta
        9. HIVE-20699.8.patch
          61 kB
          Vaibhav Gumashta
        10. HIVE-20699.9.patch
          61 kB
          Vaibhav Gumashta
        11. HIVE-20699.10.patch
          66 kB
          Vaibhav Gumashta
        12. HIVE-20699.11.patch
          66 kB
          Vaibhav Gumashta
        13. HIVE-20699.11.patch
          66 kB
          Vaibhav Gumashta

        Issue Links

        depends upon

        Improvement - An improvement or enhancement to an existing feature or task. HIVE-20738 Enable Delete Event filtering in VectorizedOrcAcidRowBatchReader

        • Major - Major loss of function.
        • Closed
        is blocked by

        Bug - A problem which impairs or prevents the functions of the product. ORC-437 Make acid schema checks case insensitive

        • Major - Major loss of function.
        • Closed
        is related to

        Sub-task - The sub-task of the issue HIVE-20823 Make Compactor run in a transaction

        • Major - Major loss of function.
        • Closed
        is required by

        Bug - A problem which impairs or prevents the functions of the product. HIVE-21165 ACID: pass query hint to the writers to write hive.acid.key.index

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HIVE-21164 ACID: explore how we can avoid a move step during inserts/compaction

        • Major - Major loss of function.
        • Closed
        relates to

        Sub-task - The sub-task of the issue HIVE-20934 ACID: Query based compactor for minor compaction

        • Major - Major loss of function.
        • Closed
        links to

        Web Link Review Board

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vgumashta Vaibhav Gumashta Assign to me
            ekoifman Eugene Koifman
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment