Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33828

SQL Adaptive Query Execution QA

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Closed
    • Critical
    • Resolution: Done
    • 3.2.0
    • 3.2.0
    • SQL

    Description

      Since SPARK-31412 is delivered at 3.0.0, we received and handled many JIRA issues at 3.0.x/3.1.0/3.2.0. This umbrella JIRA issue aims to enable it by default and collect all information in order to do QA for this feature in Apache Spark 3.2.0 timeframe.

      Attachments

        Issue Links

          is related to

          Umbrella - An overarching type made of sub-tasks SPARK-37063 SQL Adaptive Query Execution QA: Phase 2

          • Major - Major loss of function.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-31412 New Adaptive Query Execution in Spark SQL

          • Major - Major loss of function.
          • Closed
          1.
          		 Enable spark.sql.adaptive.enabled true by default Sub-task Resolved Dongjoon Hyun
          2.
          		 support Dynamic Partition Pruning in Adaptive Execution Sub-task Resolved Ke Jia
          3.
          		 TPCDS Q5 fails if spark.sql.adaptive.enabled=true Sub-task Resolved Takeshi Yamamuro
          4.
          		 Use the `CastSupport.cast` method in HashJoin Sub-task Closed Unassigned
          5.
          		 EXPLAIN FORMATTED doesn't show the plan for subqueries if AQE is enabled Sub-task Resolved Kousuke Saruta
          6.
          		 Fix some tests with AQE enabled Sub-task Resolved wuyi
          7.
          		 Broadcast timeout happened unexpectedly in AQE Sub-task Resolved Yu Zhong
          8.
          		 Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules Sub-task Resolved Ke Jia
          9.
          		 Regression in "operating on canonicalized plan" check in CustomShuffleReaderExec Sub-task Resolved Andy Grove
          10.
          		 Fail in fetch shuffle blocks in batch when i/o encryption is enabled. Sub-task Resolved hezuojiao
          11.
          		 Support DPP in AQE when the broadcast exchange can be reused Sub-task Resolved Ke Jia
          12.
          		 AQE columnar mismatch on exchange reuse Sub-task Resolved Andy Grove
          13.
          		 mapred.reduce.tasks should be shuffle.partitions not adaptive.coalescePartitions.initialPartitionNum Sub-task Resolved Kent Yao 2
          14.
          		 Support AQE side broadcastJoin threshold Sub-task Resolved XiDuo You
          15.
          		 Support AQE optimizer side transformUpWithPruning Sub-task Resolved XiDuo You
          16.
          		 Use origin plan if can not coalesce shuffle partition Sub-task Resolved XiDuo You
          17.
          		 Use inputPlan instead of currentPhysicalPlan to initialize logical link Sub-task Resolved XiDuo You
          18.
          		 Coalesce shuffle partition should handle empty input RDD Sub-task Resolved XiDuo You
          19.
          		 OptimizeSkewedJoin support ShuffledHashJoinExec Sub-task Resolved XiDuo You
          20.
          		 Support AQE side shuffled hash join formula Sub-task Resolved XiDuo You
          21.
          		 EXPLAIN CODEGEN does not work with AQE Sub-task Resolved Cheng Su
          22.
          		 Do not remove REPARTITION_BY_NUM shuffle if AQE is enabled Sub-task Resolved XiDuo You
          23.
          		 Enhance EliminateUnnecessaryJoin Sub-task Resolved XiDuo You
          24.
          		 UnsupportedOperationException: WholeStageCodegen (3) does not implement doExecuteBroadcast Sub-task Resolved Ke Jia
          25.
          		 QueryExecutionListener does not see any observed metrics fired before persist/cache Sub-task Resolved Tanel Kiis
          26.
          		 Add dataSize field in CoalescedPartitionSpec Sub-task Resolved XiDuo You
          27.
          		 EXPLAIN FORMATTED for AQE Sub-task Resolved Wenchen Fan
          28.
          		 Make sure partitions are not too small in AQE partition coalescing Sub-task Resolved Wenchen Fan
          29.
          		 Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles Sub-task Resolved Wenchen Fan
          30.
          		 Disable timeout for BroadcastQueryStageExec in AQE Sub-task Resolved Kent Yao 2
          31.
          		 [SQL] Regression in AQEShuffleReadExec Sub-task Resolved Andy Grove
          32.
          		 AQE Shuffle should wait for its subqueries to finish before materializing Sub-task Resolved Wenchen Fan
          33.
          		 Adaptively enable timeout for BroadcastQueryStageExec Sub-task In Progress Unassigned

          Activity

            People

              Unassigned Unassigned
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: