Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.1
-
None
-
None
Description
The challenge is whenever you do spark-submit to start the application, the jars present in spark home directory gets added to classpath automatically and there is no way to exclude specific jars from there. For example, we dont want slf4j jars present in spark home directory to be setted in classpath as in codebase slf4j is already there. Thus it causes conflicts in jars. This forces user to change there codebase to support spark-submit or to manually remove the jars from spark-home directory. This i believe is not right practice as we deviating from using spark as it supposed to be and it causes unfixable behaviors at various instances with no clue. Example linkages errors are common with the jar conflicts.
There is detailed stackoverflow question on this issue.
refer : https://stackoverflow.com/questions/76476618/linkageerror-facing-while-doing-spark-submit