Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25342

Support rolling back a result stage

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core
    • None

    Description

      This is a follow up of https://issues.apache.org/jira/browse/SPARK-23243

      To completely fix that problem, Spark needs to be able to rollback a result stage and rerun all the result tasks.

      However, the result stage may do file committing, which does not support re-commit a task currently. We should either support to rollback a committed task, or abort the entire committing and do it again.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-23243 Shuffle+Repartition on an RDD could lead to incorrect answers

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-25341 Support rolling back a shuffle map stage and re-generate the shuffle files

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #37359 [Github] Pull Request #37359 (caican00)

          Activity

            People

              Unassigned Unassigned
              cloud_fan Wenchen Fan
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: