Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45115

No way to exclude jars setting to classpath while doing spark-submit

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.1
    • None
    • Spark Submit
    • None

    Description

      The challenge is whenever you do spark-submit to start the application, the jars present in spark home directory gets added to classpath automatically and there is no way to exclude specific jars from there. For example, we dont want slf4j jars present in spark home directory to be setted in classpath as in codebase slf4j is already there. Thus it causes conflicts in jars. This forces user to change there codebase to support spark-submit or to manually remove the jars from spark-home directory. This i believe is not right practice as we deviating from using spark as it supposed to be and it causes unfixable behaviors at various instances with no clue. Example linkages errors are common with the jar conflicts. 

       

      There is detailed stackoverflow question on this issue. 

      refer : https://stackoverflow.com/questions/76476618/linkageerror-facing-while-doing-spark-submit

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            sumanto_pal_07 Sumanto Pal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 336h
                336h
                Remaining:
                Remaining Estimate - 336h
                336h
                Logged:
                Time Spent - Not Specified
                Not Specified