Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13356

Balancer:Set default value of minBlockSize to 10mb

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

       It seems we can run into a problem while a rolling upgrade with this.
      The Balancer is upgraded after NameNodes, so once NN is upgraded it will expect minBlockSize parameter in getBlocks(). The Balancer cannot send it yet, so NN will use the default, which you set to 0. So NN will start unexpectedly sending small blocks to the Balancer. So we should

      1. either change the default in protobuf to 10 MB
      2. or treat minBlockSize == 0 in NameNodeRpcServer as a signal to use the configuration variable DFSConfigKeys.DFS_BALANCER_GETBLOCKS_MIN_BLOCK_SIZE_KEY.

      If you agree, we should create a follow up jira. I wanted to backport this down the chain of branches, but this upgrade scenario is stopping me.

      shv commented this in  HDFS-13222 jira.

      https://issues.apache.org/jira/browse/HDFS-13222?focusedCommentId=16414855&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16414855

      Attachments

        1. HDFS-13356.02.patch
          0.9 kB
          Bharat Viswanadham
        2. HDFS-13356.01.patch
          0.9 kB
          Bharat Viswanadham
        3. HDFS-13356.00.patch
          0.9 kB
          Bharat Viswanadham

        Issue Links

          Activity

            People

              bharat Bharat Viswanadham
              bharat Bharat Viswanadham
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: