Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18309

Support multi threads in CleanerChore

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • None
    • 2.0.0-beta-1, 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      After HBASE-18309,
      1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir.
      2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default.
      3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does.
      4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.
      Show
      After HBASE-18309 , 1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir. 2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default. 3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does. 4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.

    Description

      There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find this is not enough. The number of files under oldWALs reach the max-directory-items limit of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash not happened any more.

      What's more, currently there's only one thread iterating the archive directory, and we could use multiple threads cleaning sub directories in parallel to speed it up.

      Attachments

        1. space_consumption_in_archive.png
          202 kB
          Yu Li
        2. HBASE-18309.master.012.patch
          33 kB
          Reid Chan
        3. HBASE-18309.master.011.patch
          33 kB
          Reid Chan
        4. HBASE-18309.master.010.patch
          32 kB
          Reid Chan
        5. HBASE-18309.master.009.patch
          32 kB
          Reid Chan
        6. HBASE-18309.master.008.patch
          32 kB
          Reid Chan
        7. HBASE-18309.master.007.patch
          32 kB
          Reid Chan
        8. HBASE-18309.master.006.patch
          32 kB
          Reid Chan
        9. HBASE-18309.master.005.patch
          32 kB
          Reid Chan
        10. HBASE-18309.master.004.patch
          31 kB
          Reid Chan
        11. HBASE-18309.master.002.patch
          16 kB
          Reid Chan
        12. HBASE-18309.master.001.patch
          15 kB
          Reid Chan
        13. HBASE-18309.branch-1.006.patch
          40 kB
          Zach York
        14. HBASE-18309.branch-1.005.patch
          40 kB
          Reid Chan
        15. HBASE-18309.branch-1.004.patch
          40 kB
          Reid Chan
        16. HBASE-18309.branch-1.003.patch
          40 kB
          Reid Chan
        17. HBASE-18309.branch-1.002.patch
          42 kB
          Reid Chan
        18. HBASE-18309.branch-1.001.patch
          42 kB
          Reid Chan
        19. HBASE-18309.addendum.patch
          1 kB
          Chia-Ping Tsai

        Issue Links

          Activity

            People

              reidchan Reid Chan
              binlijin Lijin Bin
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: