Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6331 [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
  3. MAPREDUCE-6720

Inconsistent values of counters across tasks and job reported to timeline service.

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      While testing found below issue. For some of the task counters, we do not have consistent values. This is not the case with every counter though.
      Consider the case of counter "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN".

      I found that its value for a flow I ran, was 936018 bytes. For the 3 apps associated with this flow run, the values were 312006 bytes each (which equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further down I found though that for one of the apps, the 4 tasks(2 mappers and 2 reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes each for the 2 mappers.

      This means the value reported for the app should be (2 * 156003 + 2* 155918) or 623842 bytes but it is only 312006 bytes which indicates that only counter value of mappers is being picked up.

      Attachments

        1. Counters For Job.png
          326 kB
          Varun Saxena
        2. Counters for Tasks.png
          198 kB
          Varun Saxena
        3. MAPREDUCE-6720-YARN-2928.01.patch
          1 kB
          Varun Saxena
        4. MAPREDUCE-6720-YARN-2928.02.patch
          6 kB
          Varun Saxena
        5. MAPREDUCE-6720-YARN-2928.03.patch
          6 kB
          Varun Saxena
        6. MAPREDUCE-6720-YARN-2928.04.patch
          4 kB
          Varun Saxena

        Activity

          People

            varun_saxena Varun Saxena
            varun_saxena Varun Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: