Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29070

Make SparkLauncher log full spark-submit command line

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.5
    • 3.0.0
    • Spark Submit
    • None

    Description

      org.apache.spark.launcher.SparkLauncher wraps a ProcessBuilder, and builds up a full command line to spark-submit using a builder pattern. When startApplication is finally called, a full command line is materialized out of all the options, then invoked via the ProcessBuilder.

      In scenarios where another application is submitting to Spark, it would be extremely useful from a support and debugging standpoint to be able to see the full spark-submit command that is actually used (so that the same submission can be tested standalone, arguments tweaked, etc.). Currently, the only way this gets captured is to stderr if the SPARK_PRINT_LAUNCH_COMMAND environment variable is set. This is cumbersome in the context of an application that is wrapping Spark and already using the APIs.

      I propose simply making SparkSubmit log the full command line it is about to launch, so that clients can see it directly in their log files, rather than having to capture and search through stderr.

      Attachments

        Issue Links

          Activity

            People

              jeff.w.evans Jeff Evans
              jeff.w.evans Jeff Evans
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified