Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18662

hive.acid.key.index is missing entries

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • Transactions
    • None

    Description

      OrcRecordUpdater.KeyIndexBuilder stores an index in ORC footer where each entry is the last ROW__ID of each stripe.  In acid1 this is used to filter the events from delta file when merging with part of the base.

       

      as can be seen in TestTxnCommands.testVersioning() (added in HIVE-18659) the hive.acid.key.index is empty.  

       

      This is because very little data is written and WriterImpl.flushStripe() is not called except when {

      {WriterImpl.close()}

      is called.  In the later, WriterCallback.preFooterWrite() is called before preStripeWrite and so KeyIndexBuilder.preFooterWriter() records nothing in hive.acid.key.index

       

      need to investigate if this is an issue, in particular acid 2

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue HIVE-16812 VectorizedOrcAcidRowBatchReader doesn't filter delete events

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is superceded by

          Bug - A problem which impairs or prevents the functions of the product. HIVE-18817 ArrayIndexOutOfBounds exception during read of ACID table.

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-18659 add acid version marker to acid files/directories

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-17284 remove OrcRecordUpdater.deleteEventIndexBuilder

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: