Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17090

Decommission will be stuck for long time when restart because overlapped process Register and BlockReport.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • namenode
    • None

    Description

      I met one corner case recently, which decommission DataNode impact performance of NameNode. After dig carefully, I have reproduced this case.
      a. Add some DataNodes to exclude and prepare to decommission this Datanodes.
      b. Execute bin/hdfs dfsadmin -refresh (This is optional step).
      c. Restart NameNode for upgrade or other reason before complete to decommission.
      d. All DataNodes will be trigger to register and FBR.
      e. Considering that the load of NameNode will be very high, especially 8040 CallQueue(which is used by DataNode interact with NameNode) will be full for a long time because RPC flood about register/heartbeat/FBR from DataNodes.
      f. For one decommission in-progress node, it will not complete to decommission until next FBR even all replicas of this node has been processed, because the request order register-heartbeat-(blockreport, register), and the second register could be one retry RPC request from DataNode (No more log information from DataNode to confirm), and for (blockreport, register), NameNode could process one storage then process register then process remaining storages in order(process FBR asynchronously now).
      g. Because the second register RPC, the related DataNodes will be marked unhealthy by BlockManager#isNodeHealthyForDecommissionOrMaintenance. So decommission will be stuck for long time until next FBR. Thus NameNode need to scan this DataNode at every round to check if could complete which hold the global write lock and impact performance of NameNode.

      To improve it, I think we could filter the repeated register RPC request at startup progress. Not think carefully if it will involve other risks when filter register directly. Welcome anymore discussions.

      Attachments

        Activity

          People

            hexiaoqiao Xiaoqiao He
            hexiaoqiao Xiaoqiao He
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: