Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25220

[K8S] Split out node selector config between driver and executors.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Duplicate
    • 2.4.0
    • None
    • Kubernetes, Spark Core

    Description

      Seperated node selector config option between executors and driver.

      This removes the spark.kubernetes.node.selector config option and seperates it to
      spark.kubernetes.driver.selector and spark.kubernetes.executor.selector

      to allow seperate node selectors on drivers and executors.

      My personal use case for this change is that on AWS we have cheap spotinstances that can terminate at any moment which is okay for executors but not the driver.
      With a single node selector option I am unable to use the spot instances to save costs on the executor nodes.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jonathan.weaver Jonathan A Weaver
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: