Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Made hbase pe RandomRead test for 100% memory-cache in HBase 2.5.0 before.
We found that there are many blocking regionserver handler threads, and our cpu utilisation percentage is about 50% and cannot increase anymore.
Here is the blocking stack:
RpcServer.default.FPBQ.Fifo.handler=116,port=16020" #299 daemon prio=5 os_prio=0 tid=0x0000ffff825d5000 nid=0x208e51 waiting for monitor entry [0x0000ffbe67734000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
- waiting to lock <0x0000fff75f465128> (a java.util.concurrent.ConcurrentHashMap$Node)
at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:77)
at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:82)
at org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:93)
at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:283)
at org.apache.hadoop.hbase.regionserver.HRegion.metricsUpdateForGet(HRegion.java:7401)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2667)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2567)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45945)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)