Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33005 Kubernetes GA Preparation
  3. SPARK-32067

Use unique ConfigMap name for executor pod template

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1, 3.1.0
    • 3.0.2, 3.1.0
    • Kubernetes
    • None

    Description

      THE BUG:

      The bug is reproducible by spark-submit two different apps (app1 and app2) with different executor pod templates (e.g., different labels) to K8s sequentially,  with app2 launching while app1 is still in the middle of ramping up all its executor pods. The unwanted result is that some launched executor pods of app1 end up having app2's executor pod template applied to them.

      The root cause appears to be that app1's podspec-configmap got overwritten by app2 during the overlapping launching periods because both apps use the same ConfigMap (name). This causes some app1's executor pods being ramped up after app2 is launched to be inadvertently launched with the app2's pod template. The issue can be seen as follows:

      First, after submitting app1, you get these configmaps:

      NAMESPACE    NAME                                       DATA    AGE
      default      app1-1111111111111111-driver-conf-map      1       9m46s
      default      podspec-configmap                          1       12m

      Then submit app2 while app1 is still ramping up its executors. The podspec-confimap is modified by app2.

      NAMESPACE    NAME                                       DATA    AGE
      default      app1-1111111111111111-driver-conf-map      1       11m43s
      default      app2-2222222222222222-driver-conf-map      1       10s
      default      podspec-configmap                          1       13m57s

       

      PROPOSED SOLUTION:

      Properly prefix the podspec-configmap for each submitted app, ideally the same way as the driver configmap:

      NAMESPACE    NAME                                       DATA    AGE
      default      app1-1111111111111111-driver-conf-map      1       11m43s
      default      app1-1111111111111111-podspec-configmap    1       13m57s
      default      app2-2222222222222222-driver-conf-map      1       10s 
      default      app2-2222222222222222-podspec-configmap    1       3m

      Attachments

        Activity

          People

            sdehaes Stijn De Haes
            james.yu@ymail.com James Yu
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: