Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20734

Colocate recovered edits directory with hbase.wal.dir

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      Previously the recovered.edits directory was under the root directory. This JIRA moves the recovered.edits directory to be under the hbase.wal.dir if set. It also adds a check for any recovered.edits found under the root directory for backwards compatibility. This gives improvements when a faster media(like SSD) or more local FileSystem is used for the hbase.wal.dir than the root dir.
      Show
      Previously the recovered.edits directory was under the root directory. This JIRA moves the recovered.edits directory to be under the hbase.wal.dir if set. It also adds a check for any recovered.edits found under the root directory for backwards compatibility. This gives improvements when a faster media(like SSD) or more local FileSystem is used for the hbase.wal.dir than the root dir.

    Description

      During investigation of HBASE-20723, I realized that we wouldn't get the best performance when hbase.wal.dir is configured to be on different (fast) media than hbase rootdir w.r.t. recovered edits since recovered edits directory is currently under rootdir.

      Such setup may not result in fast recovery when there is region server failover.

      This issue is to find proper (hopefully backward compatible) way in colocating recovered edits directory with hbase.wal.dir .

      Attachments

        1. HBASE-20734.branch-1.005.patch
          50 kB
          Zach York
        2. HBASE-20734.master.012.patch
          66 kB
          Zach York
        3. HBASE-20734.branch-1.004.patch
          48 kB
          Zach York
        4. HBASE-20734.master.011.patch
          65 kB
          Zach York
        5. HBASE-20734.master.010.patch
          65 kB
          Zach York
        6. HBASE-20734.master.009.patch
          65 kB
          Zach York
        7. HBASE-20734.branch-1.003.patch
          47 kB
          Zach York
        8. HBASE-20734.master.008.patch
          64 kB
          Zach York
        9. HBASE-20734.master.007.patch
          63 kB
          Zach York
        10. HBASE-20734.branch-1.002.patch
          46 kB
          Zach York
        11. HBASE-20734.master.006.patch
          63 kB
          Zach York
        12. HBASE-20734.master.005.patch
          63 kB
          Zach York
        13. HBASE-20734.master.004.patch
          64 kB
          Zach York
        14. HBASE-20734.master.003.patch
          63 kB
          Zach York
        15. HBASE-20734.master.002.patch
          63 kB
          Zach York
        16. HBASE-20734.master.001.patch
          63 kB
          Zach York
        17. HBASE-20734.branch-1.001.patch
          42 kB
          Zach York

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HBASE-22617 Recovered WAL directories not getting cleaned up

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-17437 Support specifying a WAL directory outside of the root directory

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-20723 Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21544 Backport HBASE-20734 Colocate recovered edits directory with hbase.wal.dir

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-22200 WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir

          • Major - Major loss of function.
          • Closed
          links to

          Web Link Review Board (branch-1)

          Web Link Review Board (master)

          Activity

            People

              zyork Zach York
              yuzhihong@gmail.com Ted Yu
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: