Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47208

Allow overriding base overhead memory

    XMLWordPrintableJSON

Details

    Description

      We can already select the desired overhead memory directly via the 'spark.driver/executor.memoryOverhead' flags, however, if that flag is not present the overhead memory calculation goes as follows:

      overhead_memory = Max(384, 'spark.driver/executor.memory' * 'spark.driver/executor.memoryOverheadFactor')
      
      where the 'memoryOverheadFactor' flag defaults to 0.1

      There are certain times where being able to override the 384Mb minimum directly can be beneficial. We may have a scenario where a lot of off-heap operations are performed (ex: using package managers/native compression/decompression) where we don't have a need for a large JVM heap but we may still need a signficant amount of memory in the spark node. 

      Using the 'memoryOverheadFactor' flag may not prove appropriate. Since we may not want the overhead allocation to directly scale with JVM memory, as a cost saving/resource limitation problem.

      As such, I propose the addition of a 'spark.driver/executor.minMemoryOverhead' flag, which can be used to override the 384Mib value used in the overhead calculation.

      The memory overhead calculation will now be :

      min_memory = sparkConf.get('spark.driver/executor.minMemoryOverhead').getOrElse(384)
      
      overhead_memory = Max(min_memory, 'spark.driver/executor.memory' * 'spark.driver/executor.memoryOverheadFactor')

      PR: https://github.com/apache/spark/pull/45240  

       

       

       

       

      Attachments

        Issue Links

          Activity

            People

              jbcorreia Joao Correia
              jpcorreia99 Joao Correia
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: