Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24428

Priority compaction for recently split daughter regions

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      We observe that under hotspotting conditions that splitting will proceed very slowly and the "Cannot split region due to reference files being there" log line will be logged excessively. (branch-1 based production.) This is because after a region is split it must be compacted before it can be split again. Reference files must be replaced by real HFiles, normal housekeeping performed during compaction. However if the regionserver is under excessive load, its compaction queues may become deep. The daughters of a recently split hotspotting region may themselves continue to hotspot and will rapidly need to split again. If the scheduled compaction work to remove/replace reference files is queued hundreds or thousands of compaction queue elements behind current, the recently split daughter regions will not be able to split again for a long time and may grow very large, producing additional complications (very large regions, very deep replication queues).

      To help avoid this condition we should prioritize the compaction of recently split daughter regions. Compaction requests include a priority field and CompactionRequest implements a comparator that sorts by this field. We already detect when a compaction request involves a region that has reference files, to ensure that it gets selected to be eligible for compaction, but we do not seem to prioritize the requests for post-split housekeeping. Split work should be placed at the top of the queue. Ensure that this is happening.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vjasani Viraj Jasani
            apurtell Andrew Kyle Purtell
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment