Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0-beta-1
-
None
-
Reviewed
Description
HMaster encountered a deadlock when runing ITBLL test.
The handler thread of HMaster holds the synchronized lock of SnapshotManager and competes for the readLock of takingSnapshotLock.
Thread 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000): State: WAITING Blocked count: 14 Waited count: 206 Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@a7a58d4 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:675) org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1765) org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:437) org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102) org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
The Master Snapshot cleaning thread holds the writeLock of takingSnapshotLock to compete for the synchronized lock of SnapshotManager.
Thread 5405 (hfile_cleaner-dir-scan-pool-2): State: BLOCKED Blocked count: 50535 Waited count: 92200 Blocked on org.apache.hadoop.hbase.master.snapshot.SnapshotManager@21a16019 Blocked by 57 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000) Stack: org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isTakingAnySnapshot(SnapshotManager.java:655) org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:197) org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:78) org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:375) org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$1(CleanerChore.java:483) org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$788/1121007598.act(Unknown Source) org.apache.hadoop.hbase.master.cleaner.CleanerChore.deleteAction(CleanerChore.java:565) org.apache.hadoop.hbase.master.cleaner.CleanerChore.traverseAndDelete(CleanerChore.java:483) org.apache.hadoop.hbase.master.cleaner.CleanerChore.lambda$traverseAndDelete$2(CleanerChore.java:495) org.apache.hadoop.hbase.master.cleaner.CleanerChore$$Lambda$794/1250580839.run(Unknown Source) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750){{}}
Attachments
Issue Links
- is caused by
-
HBASE-26323 Introduce a SnapshotProcedure
-
- Resolved
-
- relates to
-
HBASE-27919 Run ITBLL for branch-3
-
- Open
-
- links to