Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Currently for batch jobs, individual Spark configs are added as --conf parameter in Spark submit command line. If there are many Spark configurations added in Livy APIs, it might increase the length of command line and in some cases might cause issues due to limit on command line length (Windows has 8191 character length limit on the command line length.).
We can merge the configurations into a temporary properties file and add that as --properties-file opt in spark-submit command