Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40480

Remove push-based shuffle data after query finished

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.4.0
    • Shuffle
    • None

    Description

      Now spark will only cleanup shuffle data files except push-based shuffle files.
      In our production cluster, push-based shuffle service will create too many shuffle merge data files as there are several spark thrift server.
      Could we cleanup the merged data files after the query finished?

      Attachments

        Issue Links

          Activity

            People

              wankun Wan Kun
              wankun Wan Kun
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: