Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7168

Add option to not kill already-done map tasks when node becomes unusable

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.9.2
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None
    • Environment:

      Google Compute Engine (Dataproc), Java 8

    • Target Version/s:
    • Flags:
      Patch

      Description

      When a node becomes unusable, if there are still reduce tasks running, all completed map tasks that were run on that node are killed so that they can be re-run on a different node. This is because the node can no longer serve shuffle data, so the map task output cannot be fetched by the reducers.

      If map tasks do not write their shuffle data locally, killing already-done map tasks will make the job lose map progress unnecessarily. This change prevents map progress from being lost when shuffle data is not written locally by providing a property mapreduce.map.rerun-if-node-unusable that can be set to false to prevent killing already-done map tasks.

        Attachments

        1. MAPREDUCE-7168.patch
          10 kB
          Mikayla Konst

          Activity

            People

            • Assignee:
              mkonst Mikayla Konst
              Reporter:
              mkonst Mikayla Konst
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: