Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24926

Ensure numCores is used consistently in all netty configuration (driver and executors)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • Spark Core

    Description

      I think there may be some places where we're not passing the right number of configured cores to netty – in particular in driver mode, we're not respecting "spark.driver.cores". This means that spark will configure netty will be configured based on the number of physical cores of the device, instead of whatever resources spark requested from the cluster manager. It controls both the number of threads netty uses and the number of arenas in its memory pools.

      Attachments

        Activity

          People

            nsheth Nihar Sheth
            irashid Imran Rashid
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: