Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26934

python dependencies with "local:" URIs are not visible to executors

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.4.0
    • None
    • PySpark, Spark Core

    Description

      Noticed this when running tests for SPARK-24736. If you provide a file with a "local:" URI to --py-files, spark-submit makes it visible to the driver, but not executors. That makes code that runs in executors and references those files fail.

      The underlying issue is that when providing dependency information to executors, when running tasks, SparkContext erases any references to files with a "local:" scheme, so executors don't know they exist.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              vanzin Marcelo Masiero Vanzin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: