Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4584

Umbrella: Preemption and restart of MapReduce tasks

    XMLWordPrintableJSON

Details

    Description

      This JIRA will track the implementation of improvements to the handling of intermediate data (e.g., map output). Specifically, it tracks changes in support of preempting running tasks, checkpointing completed work, and spawning one or more tasks to complete the original split/partition. These mechanisms allow one to manage skew in intermediate data, respond to resource abundance or scarcity (particularly with preemption), speculatively execute on the remaining work from checkpointed tasks, and automatically tune parameters for performance.

      Iterations will build on learnings from previous work, including the following:

      Technical reports:
      http://research.yahoo.com/files/yl-2012-002.pdf
      http://research.yahoo.com/files/yl-2012-003.pdf

      Source code:
      http://code.google.com/p/sailfish

      Attachments

        Issue Links

          contains

          Sub-task - The sub-task of the issue YARN-567 RM changes to support preemption for FairScheduler and CapacityScheduler

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-568 FairScheduler: support for work-preserving preemption

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-569 CapacityScheduler: support for preemption (using a capacity monitor)

          • Major - Major loss of function.
          • Closed
          incorporates

          Sub-task - The sub-task of the issue YARN-650 User guide for preemption

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-5176 Preemptable annotations (to support preemption in MR)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. YARN-45 [Preemption] Scheduler feedback to AM to release containers

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-4585 Checkpoint shuffle aggregation as map output

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-4586 Reduce large output segments directly from remote host

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4587 Support fetch by key boundaries for memcmp types

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4588 Map local segments as on-disk segments

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4589 MapTask preemption

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4590 ReduceTask preemption

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4591 Extend IFile format to include optional metadata

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4592 Collect statistics on key distributions/samples in intermediate data

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5196 CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5197 Checkpoint Service: a library component to facilitate checkpoint of task state

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. MAPREDUCE-5192 Separate TCE resolution from fetch

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. MAPREDUCE-5194 Heed interrupts during Fetcher shutdown

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. YARN-291 [Umbrella] Dynamic resource configuration

          • Major - Major loss of function.
          • Open

          Activity

            People

              cdouglas Christopher Douglas
              sriramsrao Sriram Rao
              Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated: