Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-870

Provide Spark config via --properties-file for Batch Jobs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 0.9.0
    • None
    • None

    Description

      Currently for batch jobs, individual Spark configs are added as --conf parameter in Spark submit command line. If there are many Spark configurations added in Livy APIs, it might increase the length of command line and in some cases might cause issues due to limit on command line length (Windows has 8191 character length limit on the command line length.).

       

      We can merge the configurations into a temporary properties file and add that as --properties-file opt in spark-submit command

      Attachments

        Activity

          People

            Unassigned Unassigned
            akbordia Akshat Bordia
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m