Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.5.1
Description
We can already select the desired overhead memory directly via the 'spark.driver/executor.memoryOverhead' flags, however, if that flag is not present the overhead memory calculation goes as follows:
overhead_memory = Max(384, 'spark.driver/executor.memory' * 'spark.driver/executor.memoryOverheadFactor') where the 'memoryOverheadFactor' flag defaults to 0.1
There are certain times where being able to override the 384Mb minimum directly can be beneficial. We may have a scenario where a lot of off-heap operations are performed (ex: using package managers/native compression/decompression) where we don't have a need for a large JVM heap but we may still need a signficant amount of memory in the spark node.
Using the 'memoryOverheadFactor' flag may not prove appropriate. Since we may not want the overhead allocation to directly scale with JVM memory, as a cost saving/resource limitation problem.
As such, I propose the addition of a 'spark.driver/executor.minMemoryOverhead' flag, which can be used to override the 384Mib value used in the overhead calculation.
The memory overhead calculation will now be :
min_memory = sparkConf.get('spark.driver/executor.minMemoryOverhead').getOrElse(384) overhead_memory = Max(min_memory, 'spark.driver/executor.memory' * 'spark.driver/executor.memoryOverheadFactor')
PR: https://github.com/apache/spark/pull/45240
Attachments
Issue Links
- links to