Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23949

Introduce caching layer in HS2 to accelerate query compilation

    XMLWordPrintableJSON

Details

    Description

      One of the major contributors to compilation latency is the retrieval of metadata from HMS. This JIRA introduces a caching layer in HS2 for this metadata. Its design is simple, relying on snapshot information being queried to cache and invalidate the metadata. This will help us to reduce the time spent in compilation by using HS2 memory more effectively, and it will allow us to improve HMS throughput for multi-tenant workloads by reducing the number of calls it needs to serve.
      This patch only caches partition retrieval information, which is often one of the most costly metadata operations. It also sets the foundation to cache additional calls in follow-up work.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HIVE-25918 Invalid stats after multi inserting into the same partition

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HIVE-24034 Add getTable to HS2 local cache

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HIVE-24025 Add getAggrStatsFor to HS2 cache

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-24176 Create query-level cache for HMS requests and extend existing local HS2 HMS cache

          • Major - Major loss of function.
          • Closed
          Parent Feature

          Bug - A problem which impairs or prevents the functions of the product. HIVE-23979 Resolve spotbugs errors in JsonReporter.java, Metrics.java, and PerfLogger.java

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-23978 Enable logging with PerfLogger in HMS client

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link GitHub Pull Request #1317

          Activity

            People

              soumyakanti.das Soumyakanti Das
              soumyakanti.das Soumyakanti Das
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m