Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6918

ShuffleMetrics.ShuffleConnections Gauge Metric Climbs Infinitely

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      We recently noticed that the mapred.ShuffleMetrics.ShuffleConnections metric seems to climb infinitely, up to many millions (see attached graph), despite being supposedly a gauge measure of the number of open connections:

      ShuffleHandler.java
          @Metric("# of current shuffle connections")
              MutableGaugeInt shuffleConnections;
      

      It seems that shuffleConnections gets incremented once for every map fetched, but only decremented once for every request. It seems to me it should be modified to only be incremented once for every request rather than for every map fetched, but I'm not familiar with the original intent.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                xkrogen Erik Krogen
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: