Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-5564

Introduce Bloom Filter to optimize data scanning based on Spark

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 5.0-alpha
    • 5.0-beta
    • Query Engine
    • None

    Description

      Currently, all the data generated by Kylin are saved as Parquet files through Spark, but Kylin has not make full use of the features of Parquet when scanning data. Among them, BloomFilter must be stressed, because it's the most common tool to help READERs to skip useless data.

      Therefore, we introduced an approach to build BloomFilter automatically, conditionally and smartly when constructing segments, on the desired columns especially according to the query histories.

      After brought in BloomFilter, Spark will have a good performance improvement in the most cases.

       

      About the benchmarks or performance tests, please read the attached PDF is the report testing on SSB.

       

      Attachments

        Activity

          People

            newboy Guangyuan Feng
            newboy Guangyuan Feng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: