Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4992

Make balancer's thread count configurable

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0-beta
    • balancer & mover
    • None
    • Reviewed

    Description

      By default, balancer has 1000 threads which moves blocks around (mover threads) and 200 threads which decide which block to move (dispatcher threads).

      On large clusters, 1000 threads creates significant load on NN, which slows down other HDFS activity. For example, on our cluster, 'hdfs dfs -ls /' command took about 1 minute when balancer is active. When no balancing in progress, the same command finishes in second or two.

      This patch makes amount of threads configurable by two new options 'dfs.balancer.moverThreads' and 'dfs.balancer.dispatcherThreads'.

      Attachments

        1. HDFS-4992-v1.patch
          2 kB
          Max Lapan
        2. HDFS-4992-v2.patch
          3 kB
          Max Lapan

        Activity

          People

            shmuma Max Lapan
            shmuma Max Lapan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: