Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-26827

RegionServer JVM crash when compact mob table

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Compaction
    • None

    Description

      When compact a mob table, RS JVM may crash or failed to do compaction as the following logs:

      2022-03-11T16:18:44,089 ERROR [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45525-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(675): Compaction failed Request=regionName=t1,,1646986716811.964618e679a2434aa7d27018baef8154., storeName=A, fileCount=2, fileSize=2.0 M (1010.2 K, 1010.2 K), priority=1, time=1646986723135java.io.IOException: Mob compaction failed for region: 964618e679a2434aa7d27018baef8154    at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:574) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:365) ~[classes/:?]    at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.compact(DefaultMobStoreCompactor.java:225) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:125) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1141) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2442) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:656) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:702) ~[classes/:?]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]Caused by: java.io.IOException: Added a key not lexically larger than previous. Current cell = org.apache.hadoop.hbase.PrivateCellUtil$ValueAndTagRewriteByteBufferExtendedCell@565d5bac, prevCell = user00000000000000000000/A:filed01/1646986721047/Put/vlen=0/mvcc=0    at org.apache.hadoop.hbase.util.BloomContext.sanityCheck(BloomContext.java:63) ~[classes/:?]    at org.apache.hadoop.hbase.util.BloomContext.writeBloom(BloomContext.java:54) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.StoreFileWriter.appendGeneralBloomfilter(StoreFileWriter.java:296) ~[classes/:?]    at org.apache.hadoop.hbase.regionserver.StoreFileWriter.append(StoreFileWriter.java:315) ~[classes/:?]    at org.apache.hadoop.hbase.mob.DefaultMobStoreCompactor.performCompaction(DefaultMobStoreCompactor.java:464) ~[classes/:?]    ... 10 more 

      It is the same problem as HBASE-25929, because DefaultMobStoreCompactor overwrite performCompaction method of DefaultCompactor.

      Attachments

        Issue Links

          Activity

            People

              meiyi Yi Mei
              meiyi Yi Mei
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: