Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Won't Fix
-
2.4.11
-
None
-
None
Description
KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs when closing a scanning region, who is adding scanners to the delayed close list.
2022-03-18 17:24:17,435 INFO [regionserver/hostname/ip:16020-splits-0] regionserver.SplitRequest: Running rollback/cleanup of failed split of hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.; java.util.ConcurrentModificationException java.io.IOException: java.util.ConcurrentModificationException at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593) at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394) at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276) at org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567) at org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221) at org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548) at org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520) at org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296) at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001) at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131) at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745) at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more
Attachments
Issue Links
- links to