Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17626

Query reoptimization using cached runtime statistics

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Logical Optimizer

    Description

      Something similar to "EXPLAIN ANALYZE" where we annotate explain plan with actual and estimated statistics. The runtime stats can be cached at query level and subsequent execution of the same query can make use of the cached statistics from the previous run for better optimization.
      Some use cases,
      1) re-planning join query (mapjoin failures can be converted to shuffle joins)
      2) better statistics for table scan operator if dynamic partition pruning is involved
      3) Better estimates for bloom filter initialization (setting expected entries during merge)

      This can extended to support wider queries by caching fragments of operator plans scanning same table(s) or matching some operator sequences.

      Attachments

        1. runtimestats.patch
          46 kB
          Prasanth Jayachandran
        2. HIVE-17626.01wip01.patch
          143 kB
          Zoltan Haindrich
        3. HIVE-17626.01.patch
          181 kB
          Zoltan Haindrich
        4. HIVE-17626.02.patch
          268 kB
          Zoltan Haindrich
        5. HIVE-17626.03.patch
          232 kB
          Zoltan Haindrich
        6. HIVE-17626.04.patch
          248 kB
          Zoltan Haindrich
        7. HIVE-17626.05.patch
          248 kB
          Zoltan Haindrich
        8. HIVE-17626.06.patch
          301 kB
          Zoltan Haindrich
        9. HIVE-17626.07B.patch
          299 kB
          Zoltan Haindrich
        10. HIVE-17626.07A.patch
          302 kB
          Zoltan Haindrich
        11. HIVE-17626.08.patch
          303 kB
          Zoltan Haindrich
        12. HIVE-17626.09.patch
          304 kB
          Zoltan Haindrich
        13. HIVE-17626.10.patch
          304 kB
          Zoltan Haindrich
        14. HIVE-17626.11.patch
          302 kB
          Zoltan Haindrich

        Issue Links

          Activity

            People

              kgyrtkirk Zoltan Haindrich
              prasanth_j Prasanth Jayachandran
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: