Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20636

Introduce two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      Add two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED
      1. ROWPREFIX_FIXED_LENGTH: specify the length of the prefix
      2. ROWPREFIX_DELIMITED: specify the delimiter of the prefix
      Need to specify parameters for these two types of bloomfilter, otherwise the table will fail to create
      Example:
      create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_FIXED_LENGTH', CONFIGURATION => {'RowPrefixBloomFilter.prefix_length' => '10'}}
      create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_DELIMITED', CONFIGURATION => {'RowPrefixDelimitedBloomFilter.delimiter' => '#'}}
      Show
      Add two bloom filter type : ROWPREFIX_FIXED_LENGTH and ROWPREFIX_DELIMITED 1. ROWPREFIX_FIXED_LENGTH: specify the length of the prefix 2. ROWPREFIX_DELIMITED: specify the delimiter of the prefix Need to specify parameters for these two types of bloomfilter, otherwise the table will fail to create Example: create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_FIXED_LENGTH', CONFIGURATION => {'RowPrefixBloomFilter.prefix_length' => '10'}} create 't1', {NAME => 'f1', BLOOMFILTER => 'ROWPREFIX_DELIMITED', CONFIGURATION => {'RowPrefixDelimitedBloomFilter.delimiter' => '#'}}

    Description

      As we all know, HBase uses BloomFilter(ROW and ROWCOL) to filter unnecessary files to improve read performance. But they only support Get and do not support Scan.

      In our company(Tencent), many users need to scan all rows with the same prefix, such as Tencent Game. Game user's some operational record will be written into HBase, each game user will have a lot of records, the rowkey is constructed as userid+'#'+timestamps. So we can scan all records for a given user for a specified period.

      For this scenario, we designed the prefix Bloom filter. If the startRow and stopRow of the Scan has a valid common prefix, the scan will be allowed to use BloomFilter to filter files which will enhance the performance of the scan.

      Now, this feature has been running on our cluster over a year, and scan performance for this scenario has been improved by more than one times than before.

      Attachments

        1. HBASE-20636.master.001.patch
          58 kB
          Guangxu Cheng
        2. HBASE-20636.master.002.patch
          59 kB
          Guangxu Cheng
        3. HBASE-20636.master.003.patch
          65 kB
          Guangxu Cheng
        4. HBASE-20636.master.004.patch
          65 kB
          Andrew Kyle Purtell
        5. HBASE-20636.master.005.patch
          65 kB
          Andrew Kyle Purtell

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. HBASE-10342 RowKey Prefix Bloom Filter

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-21520 TestMultiColumnScanner cost long time when using ROWCOL bloom type

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HBASE-21678 Port HBASE-20636 (Introduce two bloom filter type ROWPREFIX and ROWPREFIX_DELIMITED) to branch-1

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21922 BloomContext#sanityCheck may failed when use ROWPREFIX_DELIMITED bloom filter

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link ReviewBoard(master)

          Activity

            People

              gxcheng Guangxu Cheng
              gxcheng Guangxu Cheng
              Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: