Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.3.0
-
Important
Description
Kubernetes supports a large variety of configurations to Pods. Currently only some of these are configurable from Spark, and they all operate by being plumbed from --conf arguments through to pod creation in the code.
To avoid the anti-pattern of trying to expose an unbounded Pod feature set through Spark configuration keywords, the community is interested in working out a sane way of allowing users to supply "arbitrary" Pod YAML which can be merged with the pod configurations created by the kube backend.
Multiple solutions have been considerd, including Pod Pre-sets and loading Pod template objects. A requirement is that the policy for how user-supplied YAML interacts with the configurations created by the kube back-end must be easy to reason about, and also that whatever kubernetes features the solution uses are supported on the kubernetes roadmap.
Attachments
Issue Links
- is duplicated by
-
SPARK-24434 Support user-specified driver and executor pod templates
- Resolved