Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18083

Make large/small file clean thread number configurable in HFileCleaner

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      After HBASE-18083 we could configure HFileCleaner to use multiple threads for large/small (archived) hfile cleaning with hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count, both default to 1. These properties support online configuration change.
      Show
      After HBASE-18083 we could configure HFileCleaner to use multiple threads for large/small (archived) hfile cleaning with hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count, both default to 1. These properties support online configuration change.

    Description

      Currently we have only one thread for both large and small file cleaning, but when write pressure is huge we might need more cleaner threads, so we need to make the thread number configurable.

      We observed more than 1.8PB data in archive directory online due to business access rate change, and this proposal is one of the necessary changes required.

      Default value of the configurations would still be left to 1 to keep low pressure to NN for normal case.

      Attachments

        1. HBASE-18083.patch
          8 kB
          Yu Li
        2. HBASE-18083.v2.patch
          10 kB
          Yu Li
        3. HBASE-18083.v3.patch
          13 kB
          Yu Li

        Issue Links

          Activity

            People

              liyu Yu Li
              liyu Yu Li
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: