Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39072

Fast Fail the remaining push blocks if shuffle stage finalized

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 3.3.0
    • None
    • Shuffle
    • None

    Description

      Map task will try to push all map outputs to external shuffle service now.

      After the shuffle stage is finalized, the reduce fetch blocks RPC will be blocked if there are still many map output blocks in flight.

      We could stop pushing the remaining blocks if the shuffle stage is finalized.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            wankun Wan Kun
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: