Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-27725

HBase blocking thread on java.util.concurrent.ConcurrentHashMap.computeIfAbsent

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Performance
    • None

    Description

      Made hbase pe RandomRead test for 100% memory-cache in HBase 2.5.0 before.

      We found that there are many blocking regionserver handler threads, and our cpu utilisation percentage is about 50% and cannot increase anymore.

      Here is the blocking stack:

      RpcServer.default.FPBQ.Fifo.handler=116,port=16020" #299 daemon prio=5 os_prio=0 tid=0x0000ffff825d5000 nid=0x208e51 waiting for monitor entry [0x0000ffbe67734000]

          java.lang.Thread.State: BLOCKED (on object monitor)
           at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
           - waiting to lock <0x0000fff75f465128> (a java.util.concurrent.ConcurrentHashMap$Node)
           at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:77)
           at org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:82)
           at org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:93)
           at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:283)
           at org.apache.hadoop.hbase.regionserver.HRegion.metricsUpdateForGet(HRegion.java:7401)
           at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2667)
           at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2567)
           at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45945)
           at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385)
           at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
           at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
           at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) 

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            chenfengge chenfengge
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: