Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
5.0-alpha
-
None
Description
Currently, all the data generated by Kylin are saved as Parquet files through Spark, but Kylin has not make full use of the features of Parquet when scanning data. Among them, BloomFilter must be stressed, because it's the most common tool to help READERs to skip useless data.
Therefore, we introduced an approach to build BloomFilter automatically, conditionally and smartly when constructing segments, on the desired columns especially according to the query histories.
After brought in BloomFilter, Spark will have a good performance improvement in the most cases.
About the benchmarks or performance tests, please read the attached PDF is the report testing on SSB.