Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
Reviewed
Description
BlockReportProcessingThread thread may gets exited with multiple reasons, but the current logging prints only the exception message with different stack which is difficult to debug the issue.
Existing logging:
2021-07-20 10:20:23,104 [Block report processor] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: Block report processor encountered fatal exception: java.lang.AssertionError
2021-07-20 10:20:23,104 [Block report processor] ERROR util.ExitUtil (ExitUtil.java:terminate(213)) - Terminate called
1: Block report processor encountered fatal exception: java.lang.AssertionError
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5315)
Exception in thread "Block report processor" 1: Block report processor encountered fatal exception: java.lang.AssertionError
at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5315)
Actual issue found at:
2021-07-20 10:20:23,101 [Block report processor] ERROR blockmanagement.BlockManager (BlockManager.java:run(5314)) - java.lang.AssertionError
java.lang.AssertionError
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3480)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:4280)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addBlock(BlockManager.java:4202)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:4338)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:4305)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processIncrementalBlockReport(FSNamesystem.java:4853)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$2.run(NameNodeRpcServer.java:1657)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:5334)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5312)
This issue found while working on FGL branch. But, same issue can happen in Trunk also in any error scenario.
Attachments
Issue Links
- links to