Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.4
-
None
-
None
-
pivotal kubernetes container service PKS having 3master nodes and 3 worker nodes
OS : ubuntu for k8-s cluster
Description
I'm getting below error while running spark-submit job in kubernetes , i didn't get any specific error
19/12/26 07:38:18 INFO SparkContext: Invoking stop() from shutdown hook 19/12/26 07:38:18 INFO SparkUI: Stopped Spark web UI at http://spark-ml-test-1577345808987-driver-svc.spark-jobs.svc:4040 19/12/26 07:38:18 INFO KubernetesClusterSchedulerBackend: Shutting down all executors 19/12/26 07:38:18 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down 19/12/26 07:38:18 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.) 19/12/26 07:38:18 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 19/12/26 07:38:18 INFO MemoryStore: MemoryStore cleared 19/12/26 07:38:18 INFO BlockManager: BlockManager stopped 19/12/26 07:38:18 INFO BlockManagerMaster: BlockManagerMaster stopped 19/12/26 07:38:18 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 19/12/26 07:38:18 INFO SparkContext: Successfully stopped SparkContext 19/12/26 07:38:18 INFO ShutdownHookManager: Shutdown hook called 19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory /var/data/spark-ec98f5d3-022d-4eae-87aa-c23a96c77555/spark-b757b85f-3709-4215-8208-95db840a294a/pyspark-665955c9-0dc6-49ee-9b57-0392be430f10 19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory /tmp/spark-fa047fa8-b573-44aa-ab65-e235fb0d5b09 19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory /var/data/spark-ec98f5d3-022d-4eae-87aa-c23a96c77555/spark-b757b85f-3709-4215-8208-95db840a294a