Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.0.0
-
None
-
None
Description
For Spark 4 we should aim to improve Spark's dynamic allocation. Some potential ideas here includes the following:
- Plug-gable DEA algorithms
- How to reduce wastage on the RM side? Sometimes the driver asks for some units of resources. But when RM provisions them, the driver cancels it.
- Support for "warm" executor pools which are not tied to a particular driver but start and wait for a driver to connect to them to "claim" them.
- More explicit Cost Vs AppRunTime confiugration: A good DEA algo should allow the developer to choose between cost and runtime. Sometimes developers might be ok to pay higher costs for faster execution.
- Use previous run information to inform future runs
- Better selection of executors to be scaled down
Attachments
1.
|
Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism |
|
Open | Unassigned |
2.
|
Make DEA algorithms pluggable |
|
Open | Unassigned |
3.
|
Add the option for dynamically marking containers for preemption based data |
|
Open | Unassigned |