Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30970

Fix NPE in resolving k8s master url

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.3.4, 2.4.5, 3.0.0
    • 2.4.6, 3.0.0
    • Kubernetes, Spark Core
    • None

    Description

      ```
      bin/spark-sql --master  k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4
      Exception in thread "main" java.lang.NullPointerException
      	at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739)
      	at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261)
      	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
      	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
      	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
      	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
      	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
      	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      ```
      

      Althrough k8s:///https://kubernetes.docker.internal:6443 is a wrong master url but should not throw npe

      Attachments

        Activity

          People

            Qin Yao Kent Yao 2
            Qin Yao Kent Yao 2
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: