Details
Blocker Description
From a code base that contains 1.1.2 + HBASE-14449 + HBASE-14241 and HBASE-14313 we can reproduce a dead lock with serverKilling CM easily:
Found one Java-level deadlock: ============================= "htable-pool1-t63": waiting to lock monitor 0x0000000001cb1688 (object 0x00000000806ef150, a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection), which is held by "IPC Client (1403704789) connection to enis-hbase-sep-21-6.novalocal/172.22.107.106:16020 from root" "IPC Client (1403704789) connection to enis-hbase-sep-21-6.novalocal/172.22.107.106:16020 from root": waiting to lock monitor 0x0000000001cb1738 (object 0x00000000806f0c60, a java.lang.Object), which is held by "htable-pool1-t63" Java stack information for the threads listed above: =================================================== "htable-pool1-t63": at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:819) - waiting to lock <0x00000000806ef150> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906) - locked <0x00000000806f0c60> (a java.lang.Object) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1192) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:32699) at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:129) at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:54) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:708) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "IPC Client (1403704789) connection to enis-hbase-sep-21-6.novalocal/172.22.107.106:16020 from root": at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:832) - waiting to lock <0x00000000806f0c60> (a java.lang.Object) - locked <0x00000000806ef150> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:574)