Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10436

spark-submit overwrites spark.files defaults with the job script filename

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 1.4.0
    • None
    • Spark Submit
    • Ubuntu, Spark 1.4.0 Standalone

    Description

      In my spark-defaults.conf I have configured a set of libararies to be uploaded to my Spark 1.4.0 Standalone cluster. The entry appears as:

      spark.files libarary.zip,file1.py,file2.py

      When I execute spark-submit -v test.py

      I see that spark-submit reads the defaults correctly, but that it overwrites the "spark.files" default entry and replaces it with the name if the job script, i.e. "test.py".

      This behavior doesn't seem intuitive. test.py, should be added to the spark working folder, but it should not overwrite the "spark.files" defaults.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              whisperstream axel dahl
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: