Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21975 Histogram support in cost-based optimizer
  3. SPARK-17074

generate equi-height histogram for column

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • Optimizer, SQL
    • None

    Description

      Equi-height histogram is effective in handling skewed data distribution.

      For equi-height histogram, the heights of all bins(intervals) are the same. The default number of bins we use is 254.

      Now we use a two-step method to generate an equi-height histogram:
      1. use percentile_approx to get percentiles (end points of the equi-height bin intervals);
      2. use a new aggregate function to get distinct counts in each of these bins.

      Note that this method takes two table scans. In the future we may provide other algorithms which need only one table scan.

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue SPARK-21322 support histogram in filter cardinality estimation

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-21984 Use histogram stats in join estimation

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue SPARK-18000 Aggregation function for computing bins (distinct value, count) pairs for equi-width histograms

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. SPARK-17881 Aggregation function for generating string histograms

          • Major - Major loss of function.
          • Closed
          is blocked by

          Sub-task - The sub-task of the issue SPARK-17997 Aggregation function for counting distinct values for multiple intervals

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-22100 Make percentile_approx support date/timestamp type and change the output type to be the same as input type

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #19479 [Github] Pull Request #19479 (wzhfy)

          Activity

            People

              ZenWzh Zhenhua Wang
              ron8hu Ron Hu
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: