Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20691

Storage policy should allow deferring to HDFS

    XMLWordPrintableJSON

Details

    • Incompatible change, Reviewed
    • Hide
      After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from "HOT" back to "NONE" which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be "HOT" even if you explicitly set hbase.wal.storage.policy to "HOT"
      Show
      After HBASE-20691 we have changed the default setting of hbase.wal.storage.policy from "HOT" back to "NONE" which means we defer the policy to HDFS. This fixes the problem of release 2.0.0 that the storage policy of WAL directory will defer to HDFS and may not be "HOT" even if you explicitly set hbase.wal.storage.policy to "HOT"

    Description

      In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using "NONE" as the storage policy in hbase configs.

      As described on this dev@hbase thread "WAL storage policies and interactions with Hadoop admin tools." we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). Additionally, we can't set the policy to HOT in the event that HDFS has changed the policy for a parent directory of our WALs.

      We should put back that ability. Presuming this is done by re-adopting the "NONE" placeholder variable, we need to ensure that value doesn't get passed to HDFS APIs. Since it isn't a valid storage policy attempting to use it will cause a bunch of logging churn (which will be a regression of the problem HBASE-18118 sought to fix).

      Attachments

        1. HBASE-20691.v8.patch
          14 kB
          Yu Li
        2. HBASE-20691.v7.patch
          14 kB
          Yu Li
        3. HBASE-20691.v6.patch
          14 kB
          Yu Li
        4. HBASE-20691.v5.patch
          14 kB
          Yu Li
        5. HBASE-20691.v4.patch
          11 kB
          Yu Li
        6. HBASE-20691.v3.patch
          6 kB
          Yu Li
        7. HBASE-20691.v2.patch
          4 kB
          Yu Li
        8. HBASE-20691.patch
          1 kB
          Yu Li
        9. HBASE-20691.branch-1.v2.patch
          14 kB
          Yu Li
        10. HBASE-20691.branch-1.v2.patch
          14 kB
          Yu Li
        11. HBASE-20691.branch-1.patch
          14 kB
          Yu Li

        Issue Links

          Activity

            People

              liyu Yu Li
              busbey Sean Busbey
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: