Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27053

How about allowing engineers to use a different ExecutorBackend in StandAlone mode?

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Invalid
    • 2.3.3
    • None
    • Deploy, Spark Submit
    • None

    Description

      In Standalone mode, the command for starting an Executor JVM in is hardcoded to use org.apache.spark.executor.CoarseGrainedExecutorBackend. There seems to be no way to configure the submit operation to use a custom ExecutorBackend (a subclass of CoarseGrainedExecutorBackend).

      This is very useful when engineers need to initialize things like starting a JDBC connection and Closing JDBC connection once per Executor.

      At line 103 of StandaloneSchedulerBackend.scala, why not make the fully qualified name of the executor backend class configurable? And then fall back to this default executor backend class if it's not configured.

      val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
            args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            briggxs Ross Brigoli
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: