Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-12865

enable_reload_events breaks enable_skipping_older_events by pushing lastRefreshEventId too high

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • Catalog
    • None

    Description

      When enable_reload_events is true, catalogd fires RELOAD events after REFRESH finishes reloading the table/partition. The RELOAD event id is also used to update lastRefreshEventId of the table/partition.
      https://github.com/apache/impala/blob/784971c018c2dc44c53d7c0f366ad49cd8681ac6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6807-L6814

      This is problematic when enable_skipping_older_events is true. HMS events generated after the reload and before the RELOAD event will be skipped.
      https://github.com/apache/impala/blob/784971c018c2dc44c53d7c0f366ad49cd8681ac6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1219-L1221

      The issue is identified in the review of https://gerrit.cloudera.org/c/21096/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#6799

      CC hemanth619, VenuReddy, csringhofer

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-11535 Skip events happen before manual REFRESH

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-11822 Optimize the Refresh/Invalidate event processing by skipping unnecessary events

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              hemanth619 Sai Hemanth Gantasala
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: