Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Invalid
-
2.3.3
-
None
-
None
Description
In Standalone mode, the command for starting an Executor JVM in is hardcoded to use org.apache.spark.executor.CoarseGrainedExecutorBackend. There seems to be no way to configure the submit operation to use a custom ExecutorBackend (a subclass of CoarseGrainedExecutorBackend).
This is very useful when engineers need to initialize things like starting a JDBC connection and Closing JDBC connection once per Executor.
At line 103 of StandaloneSchedulerBackend.scala, why not make the fully qualified name of the executor backend class configurable? And then fall back to this default executor backend class if it's not configured.
val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)