Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28022

k8s pod affinity to achieve cloud native friendly autoscaling

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 3.0.0
    • None
    • Kubernetes, Spark Core
    • None

    Description

      Hi, in order to achieve cloud native friendly autoscaling , I propose to add a pod affinity feature.

      Traditionally, when we use spark in fix size yarn cluster, it make sense to spread containers to every node.

      Coming to cloud native resource manage, we want to release node when we don't need it any more.

      Pod affinity feature counts to place all pods of certain application to some nodes instead of all nodes.

      By the way,  using pod template is not a good choice, adding application id  to pod affinity term when submit is more robust.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Andrew HUALI Henry Yu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: