Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12758

Event Processor is ignoring the prev_id while reloading the existing partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Catalog

    Description

      Partitioned table Insert events consumed by the event processor reloads the partitions to update file metadata. Currently while reloading the filemetadata, 'prev_id' of the old partitions is being ignored in the partition builder

      HdfsPartition oldPartition = entry.getValue();
      HdfsPartition.Builder partBuilder = createPartitionBuilder(
         hmsPartition.getSd(), hmsPartition, permissionCache); 

      As a result 'prev_id' of the partBuilder will always be -1, so when the catalogDelta is sent from the state store to impala demons, because prev_id is not valid, impalads will not know whether to invalidate the current partition and then request the new partition information. 

      This might lead to data correctness issues.

      Attachments

        Activity

          People

            hemanth619 Sai Hemanth Gantasala
            hemanth619 Sai Hemanth Gantasala
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: