Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
Description
// If the task succeeded and the stage generates shuffle data, record that this executor // holds data for the shuffle. This code will track all executors that generate shuffle // for the stage, even if speculative tasks generate duplicate shuffle data and end up // being ignored by the map output tracker. // // This means that an executor may be marked as having shuffle data, and thus prevented // from being removed, even though the data may not be used. if (shuffleTrackingEnabled && event.reason == Success) { stageToShuffleID.get(event.stageId).foreach { shuffleId => exec.addShuffle(shuffleId) } }
Attachments
Issue Links
- relates to
-
SPARK-27963 Allow dynamic allocation without an external shuffle service
- Resolved