Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-11409

Skip UpdateCatalogMetrics if another thead is on-going in it

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Won't Do
    • None
    • None
    • Backend
    • None
    • ghx-label-10

    Description

      Impala coordinator tracks local metrics of the catalog, e.g. number of dbs/tables. When use_local_catalog is enabled, it also tracks the cache metrics, e.g. cache hit/miss count/rate.

      These metrics are updated at the end of each statement, even for simple statements like "USE <db>", "SET var=xxx", "SELECT 1". The catalog update thread will also update the metrics in the end.
      https://github.com/apache/impala/blob/bb610dee09a8069bb993b4c668f7e481c1774b70/be/src/service/impala-server.cc#L1272
      https://github.com/apache/impala/blob/bb610dee09a8069bb993b4c668f7e481c1774b70/be/src/service/impala-server.cc#L2065

      These metrics are global metrics of the local catalog cache. They are not specifit to a single statement. It's a waste to update the metrics concurrently.
      https://github.com/apache/impala/blob/bb610dee09a8069bb993b4c668f7e481c1774b70/be/src/service/impala-server.cc#L1526-L1559

      We've seen "hanging issues" that all statements, including the catalog update thread, are slowly executing the UpdateCatalogMetrics() function. See details in the attached jstack dump.

      Indeed, if one thread is running the UpdateCatalogMetrics() function, the other threads can skip it and move forward.

      Attachments

        1. jstack-1.txt
          232 kB
          Quanlong Huang

        Issue Links

          Activity

            People

              stigahuang Quanlong Huang
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: