Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
3.0.0
-
None
-
None
Description
Hi, in order to achieve cloud native friendly autoscaling , I propose to add a pod affinity feature.
Traditionally, when we use spark in fix size yarn cluster, it make sense to spread containers to every node.
Coming to cloud native resource manage, we want to release node when we don't need it any more.
Pod affinity feature counts to place all pods of certain application to some nodes instead of all nodes.
By the way, using pod template is not a good choice, adding application id to pod affinity term when submit is more robust.
Attachments
Issue Links
- relates to
-
SPARK-20628 Keep track of nodes which are going to be shut down & avoid scheduling new tasks
- Resolved