Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15020

NNBench not support run more than one map task on the same host

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 2.7.2
    • None
    • benchmarks
    • None
    • Hadoop 2.7.2

    Description

      When benchmark NameNode performance with NNBench, I start with pseudo distributed deploy. Everything goes well with "-maps 1". BUT with -maps N (n>1) and -operation create_write, many exceptions meet during the benchmark.

      Hostname is part of the file path, which can differentiate hosts. With more than two map tasks run on the same host, more than two map tasks may operate on the same file, leading to exceptions.

      17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #1: 84
      17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: AL Total #2: 43
      17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 2570
      17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 814.0
      17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: Late maps: 0
      17/11/07 15:22:32 INFO hdfs.NNBench: RAW DATA: # of exceptions: 3000

      2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded in op: Create/Write/Close
      2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded in op: Create/Write/Close
      2017-11-07 14:54:08,083 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded in op: Create/Write/Close

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              zhoutai.zt zhoutai.zt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: