Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24435

Support user-supplied YAML that can be merged with k8s pod descriptions

    XMLWordPrintableJSON

Details

    • Important

    Description

      Kubernetes supports a large variety of configurations to Pods. Currently only some of these are configurable from Spark, and they all operate by being plumbed from --conf arguments through to pod creation in the code.

      To avoid the anti-pattern of trying to expose an unbounded Pod feature set through Spark configuration keywords, the community is interested in working out a sane way of allowing users to supply "arbitrary" Pod YAML which can be merged with the pod configurations created by the kube backend.

      Multiple solutions have been considerd, including Pod Pre-sets and loading Pod template objects.  A requirement is that the policy for how user-supplied YAML interacts with the configurations created by the kube back-end must be easy to reason about, and also that whatever kubernetes features the solution uses are supported on the kubernetes roadmap.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              eje Erik Erlandson
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: