Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24417 Build and Run Spark on JDK11
  3. SPARK-27704

Change default class loader to ParallelGC

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • 3.0.0
    • None
    • Build
    • None

    Description

      In JDK 11 the default class loader changed from ParallelGC to G1GC. Even though this gc performs better on pause times and interactivity, most of the tasks that need to be processed are more sensitive to throughput and the to the amount of memory. G1 sacrifices these to some extend to avoid the big pauses. As a result the user may perceive a regression compared to JDK 8. Even worse, the regression may not be limited to performance only but some jobs may start failing in case they do not fit into the memory they used to be happy with when running with previous JDK.

      Some other kind of apps, like streaming ones, may rather use G1 because of their more interactive, more realtime needs.

      With this jira it is proposed to have a configurable default GC for all spark applications. This may be overridable by the user through command line parameters. The default value of the default GC (in case it is not provided in spark-defaults.conf) could be ParallelGC.

      I do not see this change required but I think it would benefit to the user experience.

      Attachments

        Activity

          People

            Unassigned Unassigned
            misutoth Mihaly Toth
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: