Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26320

Separate Log Cleaner DirScanPool to prevent the OLDWALs from filling up the disk when archive is large

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.7.1, 2.4.6
    • 2.5.0, 3.0.0-alpha-3, 2.4.16
    • Operability
    • None

    Description

      We currently share the DirScanPool (threadpool for scanning for files to delete in the OldLogs and archive directories) between the LogCleaner and HFileCleaner. This means that if the archive directory is large/has lots of files/directories, the threads can get stuck scanning through the archive directory, starving the LogCleaner. This is especially apparent on S3 where list can be slower than on HDFS.

      This JIRA creates separate DirScanPools for the LogCleaner and HFileCleaner

      Attachments

        Activity

          People

            zyork Zach York
            zyork Zach York
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: