Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28845

Enable spark.sql.execution.sortBeforeRepartition only for retried stages

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • 3.0.0
    • None
    • Spark Core, SQL
    • None

    Description

      For fixing the correctness bug of SPARK-28699, we disable radix sort for the scenario of repartition in Spark SQL. This will cause a performance regression.

      So for limiting the performance overhead, we'll do the optimizing work by only enable sort for the repartition operation while stage retries happening. This work depends on SPARK-25341.

      Attachments

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-25341 Support rolling back a shuffle map stage and re-generate the shuffle files

          • Major - Major loss of function.
          • Resolved
          is caused by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-28699 Cache an indeterminate RDD could lead to incorrect result while stage rerun

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              Unassigned Unassigned
              XuanYuan Yuanjian Li
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: