Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0
-
None
Description
In the current speculative execution implementation if there is only one task in the stage then no speculative run would be conducted. However, there might be cases where an executor have some problem in writing to its disk and just hang forever. In this case, if the single task stage get assigned to the problematic executor then the whole job would hang forever. It would be better if we could run the task on another executor if this happens.
Attachments
Issue Links
- is related to
-
SPARK-30417 SPARK-29976 calculation of slots wrong for Standalone Mode
- Resolved
- links to