Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.0.0
Description
Currently Spark supports migration of shuffle data to peer nodes during node decommissioning. If peer nodes are not accessible, then Spark falls back to external storage. User needs to provide the storage location path. There are scenarios where user may want to migrate to external storage instead of peer nodes. This may be because of unstable nodes or due to the need of aggressive scale down. So user should be able to configure to migrate the shuffle data directly to external storage if the use case permits.
Attachments
Issue Links
- relates to
-
SPARK-33545 Support Fallback Storage during Worker decommission
- Resolved
- links to