Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-38092

Check if shuffleMergeId is the same as the current stage's shuffleMergeId before registering MergeStatus

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.1
    • None
    • Shuffle
    • None

    Description

      Currently we have handled this in the handleShuffleMergeFinalized during finalization ensuring the finalize request is indeed for the current stage's shuffle dependency shuffleMergeId. The same check has to be done before registering merge statuses as well.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            vsowrirajan Venkata krishnan Sowrirajan

            Dates

              Created:
              Updated:

              Slack

                Issue deployment