Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Incomplete
-
2.4.0
-
None
Description
Noticed this when running tests for SPARK-24736. If you provide a file with a "local:" URI to --py-files, spark-submit makes it visible to the driver, but not executors. That makes code that runs in executors and references those files fail.
The underlying issue is that when providing dependency information to executors, when running tasks, SparkContext erases any references to files with a "local:" scheme, so executors don't know they exist.
Attachments
Issue Links
- relates to
-
SPARK-24736 --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.
- Resolved