Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.0.0
-
None
Description
After SPARK-13723, the behavior changed for spark.dynamicAllocation.initialExecutors less then spark.dynamicAllocation.minExecutors situation.
initialExecutors < minExecutors is an invalid setting,
Before SPARK-13723
If initialExecutors < minExecutors, Spark will throw exception with:
java.lang.IllegalArgumentException: requirement failed: initial executor number xxx must between min executor number xxx and max executor number xxx
This will clearly let user know that current configuration is invalid.
After SPARK-13723
Because we also consider spark.executor.instances, so the initial number is the max value between minExecutors, initialExecutors, numExecutors.
This will silently ignore the situation where initialExecutors < minExecutors.
So at least we should add some warning logs to let user know this is an invalid configuration.