Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26365

Limit min size of stripe and L0 files compaction

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 3.0.0-alpha-1, 2.0.0
    • None
    • Compaction
    • None

    Description

      In the StripeCompactPolicy#needsSingleStripeCompaction, it only checks if the file count is up to the min compaction limit.

      protected boolean needsSingleStripeCompaction(StripeInformationProvider si) {
        int minFiles = this.config.getStripeCompactMinFiles();
        for (List<HStoreFile> stripe : si.getStripes()) {
          if (stripe.size() >= minFiles) return true;
        }
        return false;
      }
      

      When select files in stripe by ExploringCompactionPolicy#applyCompactionPolicy, there is no min size limit too.

      // Sanity checks
      if (potentialMatchFiles.size() < minFiles) {
        continue;
      }
      if (potentialMatchFiles.size() > maxFiles) {
        continue;
      }
      
      // Compute the total size of files that will
      // have to be read if this set of files is compacted.
      long size = getTotalStoreSize(potentialMatchFiles);
      
      // Store the smallest set of files.  This stored set of files will be used
      // if it looks like the algorithm is stuck.
      if (mightBeStuck && size < smallestSize) {
        smallest = potentialMatchFiles;
        smallestSize = size;
      }
      
      if (size > comConf.getMaxCompactSize(mayUseOffPeak)) {
        continue;
      }
      
      ++opts;
      if (size >= comConf.getMinCompactSize()
          && !filesInRatio(potentialMatchFiles, currentRatio)) {
        continue;
      }

      While we should also limit the min size of compaction for stripes, because small stripe compactions will occupy the compaction queue.

       

      Attachments

        Issue Links

          Activity

            People

              Xiaolin Ha Xiaolin Ha
              Xiaolin Ha Xiaolin Ha
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: