Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2221

Spark fails on windows YARN (HortonWorks 2.4)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.0.0
    • None
    • YARN
    • None
    • Windows 2008 R2
      HortonWorks Hadoop cluster 2.4
      Spark 1.0

    Description

      Windows based Yarn cluster failed to execute submitted Spark job.
      Can be reproduced by submitting JavaSparkPi on Yarn cluster.

      C:\hdp\spark-1.0.0-bin-hadoop2\bin>spark-submit --class org.apache.spark.examples.JavaSparkPi ./../lib/spark-examples-1.0.0-hadoop2.2.0.jar --master yarn-cluster --deploy-mode cluster

      The origin of the problems is org.apache.spark.deploy.yarn.ExecutorRunnableUtil.scala. JavaOpts parameters that contain environment variable need to be quoted.

      • '%' should be escaped like "kill %%p" . Still doesn't have any sense on Windows. Ideally should be passed through "spark.executor.extraJavaOptions".
      • javaOpts += "-Djava.io.tmpdir=\"" + new Path(Environment.PWD.$(), YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR) + "\""

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              vrabenok Vlad Rabenok
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: