Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16138

BlockReportProcessingThread exit doesn't print the actual stack

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      BlockReportProcessingThread thread may gets exited with multiple reasons, but the current logging prints only the exception message with different stack which is difficult to debug the issue.

       

      Existing logging:

      2021-07-20 10:20:23,104 [Block report processor] INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: Block report processor encountered fatal exception: java.lang.AssertionError

      2021-07-20 10:20:23,104 [Block report processor] ERROR util.ExitUtil (ExitUtil.java:terminate(213)) - Terminate called

      1: Block report processor encountered fatal exception: java.lang.AssertionError

                      at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5315)

      Exception in thread "Block report processor" 1: Block report processor encountered fatal exception: java.lang.AssertionError

                      at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5315)

       

      Actual issue found at:

      2021-07-20 10:20:23,101 [Block report processor] ERROR blockmanagement.BlockManager (BlockManager.java:run(5314)) - java.lang.AssertionError

      java.lang.AssertionError

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addStoredBlock(BlockManager.java:3480)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processAndHandleReportedBlock(BlockManager.java:4280)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addBlock(BlockManager.java:4202)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:4338)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:4305)

                      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processIncrementalBlockReport(FSNamesystem.java:4853)

                      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer$2.run(NameNodeRpcServer.java:1657)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:5334)

                      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:5312)

       

      This issue found while working on FGL branch. But, same issue can happen in Trunk also in any error scenario.

       

      hemanthboyina hexiaoqiao

      Attachments

        Issue Links

          Activity

            People

              prasad-acit Renukaprasad C
              prasad-acit Renukaprasad C
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m