XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: HFile, regionserver
    • Labels:
      None
    • Environment:

      hadoop-2.6.0

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      After HBASE-14061 we support to set storage policy for HFile through "hbase.hstore.block.storage.policy" configuration, and we support CF-level setting to override the settings from configuration file. Currently supported storage policies include ALL_SSD/ONE_SSD/HOT/WARM/COLD, refer to http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html for more details

      For example, to create a table with two families: "cf1" with "ALL_SSD" storage policy and "cf2" with "ONE_SSD", we could use below command in hbase shell:
      create 'table',{NAME=>'f1',STORAGE_POLICY=>'ALL_SSD'},{NAME=>'f2',STORAGE_POLICY=>'ONE_SSD'}

      We could also set the configuration in table attribute like all other configurations:
      create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ONE_SSD'}}
      Show
      After HBASE-14061 we support to set storage policy for HFile through "hbase.hstore.block.storage.policy" configuration, and we support CF-level setting to override the settings from configuration file. Currently supported storage policies include ALL_SSD/ONE_SSD/HOT/WARM/COLD, refer to http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html for more details For example, to create a table with two families: "cf1" with "ALL_SSD" storage policy and "cf2" with "ONE_SSD", we could use below command in hbase shell: create 'table',{NAME=>'f1',STORAGE_POLICY=>'ALL_SSD'},{NAME=>'f2',STORAGE_POLICY=>'ONE_SSD'} We could also set the configuration in table attribute like all other configurations: create 'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ONE_SSD'}}

      Description

      After reading HBASE-12848 and HBASE-12934, I wrote a patch to implement cf-level storage policy.
      My main purpose is to improve random-read performance for some really hot data, which usually locates in certain column family of a big table.

      Usage:
      $ hbase shell
      > alter 'TABLE_NAME', METADATA =>

      {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}

      > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}

      HDFS's setStoragePolicy can only take effect when new hfile is created in a configured directory, so I had to make sub directories(for each cf) in region's .tmp directory and set storage policy for them.

      Besides, I had to upgrade hadoop version to 2.6.0 because dfs.getStoragePolicy cannot be easily written in reflection, and I needed this api to finish my unit test.

        Attachments

        1. HBASE-14061.addendum2.patch
          0.8 kB
          Yu Li
        2. HBASE-14061.addendum2.patch
          0.8 kB
          Yu Li
        3. HBASE-14061.addendum.patch
          19 kB
          Yu Li
        4. HBASE-14061.addendum.patch
          19 kB
          Yu Li
        5. HBASE-14061.v4.patch
          23 kB
          Yu Li
        6. HBASE-14061.v3.patch
          23 kB
          Yu Li
        7. HBASE-14061.v2.patch
          18 kB
          Yu Li
        8. HBASE-14061-master-v1.patch
          18 kB
          Victor Xu

          Issue Links

            Activity

              People

              • Assignee:
                liyu Yu Li
                Reporter:
                victorunique Victor Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: