Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26933

spark-submit does not make zip files provided with --py-files visible to pyspark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 2.4.0
    • None
    • PySpark, Spark Core
    • None

    Description

      I ran into this while testing the changes I made for SPARK-24736.

      If you provide, e.g., --py-files dependency.zip to spark-submit, that archive is not added to the pyspark app's PYTHONPATH. That makes the dependency not visible to the pyspark code.

      .py files added using --py-files work fine (if they're imported after a SparkContext is instantiated).

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-24736 --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              vanzin Marcelo Masiero Vanzin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: