Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47148

Avoid to materialize AQE ExchangeQueryStageExec on the cancellation

    XMLWordPrintableJSON

Details

    Description

      AQE can materialize both ShuffleQueryStage and BroadcastQueryStage on the cancellation. This causes unnecessary stage materialization by submitting Shuffle Job and Broadcast Job. Under normal circumstances, if the stage is already non-materialized (a.k.a ShuffleQueryStage.shuffleFuture or BroadcastQueryStage.broadcastFuture is not initialized yet), it should just be skipped without materializing it.

      Please find sample use-case:
      1- Stage Materialization Steps:
      When stage materialization is failed:

      1.1- ShuffleQueryStage1 - is materialized successfully,
1.2- ShuffleQueryStage2 - materialization is failed,
1.3- ShuffleQueryStage3 - Not materialized yet so ShuffleQueryStage3.shuffleFuture is not initialized yet

      2- Stage Cancellation Steps:

      2.1- ShuffleQueryStage1 - is canceled due to already materialized,
2.2- ShuffleQueryStage2 - is earlyFailedStage so currently, it is skipped as default by AQE because it could not be materialized,
2.3- ShuffleQueryStage3 - Problem is here: This stage is not materialized yet but currently, it is also tried to cancel and this stage requires to be materialized first.

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #45234

          Activity

            People

              erenavsarogullari Eren Avsarogullari
              erenavsarogullari Eren Avsarogullari
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: