Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13157

Do Not Remove Blocks Sequentially During Decommission

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • datanode, namenode
    • None

    Description

      From what I understand of DataNode decommissioning it appears that all the blocks are scheduled for removal in order.. I'm not 100% sure what the ordering is exactly, but I think it loops through each data volume and schedules each block to be replicated elsewhere. The net affect is that during a decommission, all of the DataNode transfer threads slam on a single volume until it is cleaned out. At which point, they all slam on the next volume, etc.

      Please randomize the block list so that there is a more even distribution across all volumes when decommissioning a node.

      Attachments

        1. HDFS-13157.1.patch
          7 kB
          David Mollitor

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            belugabehr David Mollitor
            belugabehr David Mollitor

            Dates

              Created:
              Updated:

              Slack

                Issue deployment