Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.0.0
-
None
-
None
Description
Spark 3.0 - Running Spark Submit as below and point to a MinKube cluster
bin/spark-submit \ --master k8s://https://192.168.99.102:8443 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.kubernetes.driver.podTemplateFile=../driver_1E.template \ --conf spark.kubernetes.executor.podTemplateFile=../executor.template \ --conf spark.kubernetes.container.image=spark:spark3 \ local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-preview2.jar 1
Spark Binaries - spark-3.0.0-preview2-bin-hadoop2.7.tgz
Driver Template -
apiVersion: v1 kind: Pod metadata: labels: spark-app-id: my-custom-id annotations: spark-driver-cpu: 1 spark-driver-mem: 1 spark-executor-cpu: 1 spark-executor-mem: 1 spark-executor-count: 1 spec: schedulerName: spark-scheduler
Executor Template
apiVersion: v1 kind: Pod metadata: labels: spark-app-id: my-custom-id spec: schedulerName: spark-scheduler
Kubernetes Pods Launched - Two Executor Pods were launched which was default
spark-pi-e608e7718f11cc69-driver 1/1 Running 0 10s spark-pi-e608e7718f11cc69-exec-1 1/1 Running 0 5s spark-pi-e608e7718f11cc69-exec-2 1/1 Running 0 5s