Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28673

Failed to start executors for spark-sql-perf on kubernetes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.3
    • None
    • Kubernetes, Spark Core, SQL
    • None

    Description

      Start Spark SQL Perf test with following command, the `spark.executor.instances` is 10, but no executors was start

       

      spark-submit \
--master k8s://https://XXX \
--deploy-mode cluster \
--name k82cn-spark-test-per \
--conf "spark.local.dir=/myfs/tmp" \
--conf "spark.executor.extraJavaOptions=-Djava.io.tmpdir=/myfs/tmp" \
--conf "spark.driver.extraJavaOptions=-Djava.io.tmpdir=/myfs/tmp" \
--class org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark \
--conf spark.kubernetes.volcano.enable=false \
--conf spark.driver.memory=3G \
--conf spark.executor.instances=10 \
--conf spark.kubernetes.container.image.pullSecrets=default-secret \
--conf spark.kubernetes.container.image=XXX/spark-test:origin-4 \
local:///opt/spark/jars/spark-sql_2.11-2.4.4-SNAPSHOT-tests.jar \
--data-location file://XXXXX/spark-tpcds-data --query-filter "q2,q5"

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            k82cn Klaus Ma
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: