Description
The spark.sql.extensions configuration options should support multiple extensions. It is currently possible to load multiple extensions using the programatic interface (e.g. SparkSession.builder().master("..").withExtensions(sparkSessionExtensions1).withExtensions(sparkSessionExtensions2).getOrCreate() ) but the same cannot currently be done with the command line options without writing a wrapper extensions that combines multiple extensions.
Allowing multiple spark.sql.extensions, would allow the extensions to be easily changes on the command line or via the configuration file. Multiple extensions could be specified using a comma separated list of class names. Allowing multiple extensions should maintain backwards compatibility because existing spark.sql.extensions configuration settings shouldn't contain a comma because the value is a class name.
Attachments
Issue Links
- links to