Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16435

Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • Scheduler, Spark Core
    • None

    Description

      After SPARK-13723, the behavior changed for spark.dynamicAllocation.initialExecutors less then spark.dynamicAllocation.minExecutors situation.

      initialExecutors < minExecutors is an invalid setting,

      Before SPARK-13723

      If initialExecutors < minExecutors, Spark will throw exception with:

      java.lang.IllegalArgumentException: requirement failed: initial executor number xxx must between min executor number xxx and max executor number xxx
      

      This will clearly let user know that current configuration is invalid.

      After SPARK-13723

      Because we also consider spark.executor.instances, so the initial number is the max value between minExecutors, initialExecutors, numExecutors.

      This will silently ignore the situation where initialExecutors < minExecutors.

      So at least we should add some warning logs to let user know this is an invalid configuration.

      What do you think tgraves, rdblue

      Attachments

        Activity

          People

            jerryshao Saisai Shao
            jerryshao Saisai Shao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: