Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47141

Support enabling migration of shuffle data directly to external storage using config parameter

    XMLWordPrintableJSON

Details

    Description

      Currently Spark supports migration of shuffle data to peer nodes during node decommissioning. If peer nodes are not accessible, then Spark falls back to external storage. User needs to provide the storage location path. There are scenarios where user may want to migrate to external storage instead of peer nodes. This may be because of unstable  nodes or due to the need of aggressive scale down. So user should be able to configure to migrate the shuffle data directly to external storage if the use case permits. 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              maheshk114 mahesh kumar behera
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: