Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-8920

Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDFS-7285
    • None
    • None
    • Reviewed

    Description

      When we test reading data with datanodes killed, DFSInputStream::getBestNodeDNAddrPair becomes a hot spot method and effectively blocks the client JVM. This log seems too verbose:

          if (chosenNode == null) {
            DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
                " after checking nodes = " + Arrays.toString(nodes) +
                ", ignoredNodes = " + ignoredNodes);
            return null;
          }
      

      Attachments

        1. HDFS-8920-HDFS-7285.2.patch
          4 kB
          Rui Li
        2. HDFS-8920-HDFS-7285.1.patch
          4 kB
          Rui Li

        Activity

          People

            lirui Rui Li
            lirui Rui Li
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: