Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
We had a situation where an hmaster had just started up and something started calling Admin.getClusterMetrics. It seems there were ServerTasks in memory, but at least one had no status. This resulted in a spam of NPE:
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=60000] ERROR org.apache.hadoop.hbase.ipc.RpcServer: Unexpected throwable object
java.lang.NullPointerException
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos$ServerTask$Builder.setStatus(ClusterStatusProtos.java:14352)
at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toServerTask(ProtobufUtil.java:3565)
at org.apache.hadoop.hbase.ClusterMetricsBuilder.lambda$toClusterStatus$4(ClusterMetricsBuilder.java:80)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.apache.hadoop.hbase.ClusterMetricsBuilder.toClusterStatus(ClusterMetricsBuilder.java:80)
at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:980)
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:385)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Attachments
Issue Links
- links to