Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-35536

Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Shuffle, Spark Core
    • None

    Description

      Currently we are not handling the case of stage cancellation and if the stage is cancelled while the stage is about to get merge finalized.

      Stage completion will wait for shuffle merge finalize wait time so that all the shuffle merge requests are drained but during this time the stage can be cancelled, if this happens then we need to let all the shuffle services to remove the merged shuffle data for the corresponding shuffle ID.

      Attachments

        Activity

          People

            Unassigned Unassigned
            vsowrirajan Venkata krishnan Sowrirajan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: