Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-28257

Memstore flushRequest can be blocked by a delayed flush scheduled by PeriodicMemstoreFlusher, RegionTooBusyExceptions for up to 5 minutes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 2.0.0, 3.0.0
    • None
    • regionserver
    • None

    Description

      Steps to reproduce:

      1. Make an edit to a region.
      2. Wait 1 hour + 10 seconds (default value of hbase.regionserver.optionalcacheflushinterval plus hbase.regionserver.flush.check.period.)
      3. Make a very large number of edits to the region (i.e. >= 1GB, pressure the memstore.)

      Expected:
      Memstore pressure leads to flushes.

      Result:
      The PeriodicMemstoreFlusher has scheduled a refresh with a random delay of 0-5 minutes (default for hbase.regionserver.periodicmemstoreflusher.rangeofdelayseconds.) Memstore pressure flushes are blocked by the scheduled delayed flush. Client receives 0-5 minutes of RegionTooBusyExceptions until the delayed flush executes.

      Logs:
      2023-12-13 06:00:13,573 INFO regionserver.HRegionServer - MemstoreFlusherChore requesting flush of <REGION> because <CF> has an old edit so flush to free WALs after random delay 166761 ms
      2023-12-13 06:00:53,219 DEBUG regionserver.HRegion - Flush requested on <REGION>
      2023-12-13 06:01:47,694 WARN regionserver.HRegion - Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G, regionName=<REGION>, server=<SERVER>
      at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
      ...
      repeats
      ...
      2023-12-13 06:01:52,223 WARN regionserver.HRegion - Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G, regionName=<REGION>, server=<SERVER>
      at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
      ...
      2023-12-13 06:03:00,340 INFO regionserver.HRegion - Flushing <REGION> 1/1 column families, dataSize=534.77 MB heapSize=1.00 GB
      2023-12-13 06:03:09,634 INFO regionserver.HRegion - Finished flush of dataSize ~534.77 MB/560744948, heapSize ~1.00 GB/1073816296, currentSize=0 B/0 for <REGION> in 9294ms, sequenceid=21310753, compaction requested=false

      Note also this is the same cause as discussed in HBASE-16030 conversation https://issues.apache.org/jira/browse/HBASE-16030?focusedCommentId=15340153&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15340153

      Attachments

        Issue Links

          Activity

            People

              dmanning David Manning
              dmanning David Manning
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: