Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12895

REFRESH doesn't detect changes in partition locations in ACID tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Catalog
    • None
    • ghx-label-11

    Description

      This was discovered by running test metadata.test_event_processing.TestEventProcessing.test_transact_partition_location_change_from_hive when flag hms_event_incremental_refresh_transactional_table  is set to false.

      https://github.com/apache/impala/blob/ab6c9467f6347671b971dbce4c640bea032b6ed9/tests/metadata/test_event_processing.py#L164

       

      When hms_event_incremental_refresh_transactional_table  is true (default), the alter partition event is processed correctly and the location change is detected. But if it is false or event processing is turned off, the change is not detected and running REFRESH on the table also doesn't update the location.

      The different handling based on the flag seems intentional:

      https://github.com/apache/impala/blob/ab6c9467f6347671b971dbce4c640bea032b6ed9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L2606

       

      This seems to be an old issues while the test was added in a recent commit:

      https://github.com/apache/impala/commit/32b29ff36fb3e05fd620a6714de88805052d0117

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              csringhofer Csaba Ringhofer
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: