Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30945

Preserve JVM information when executor heartbeat timeout

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Minor
    • Resolution: Done
    • 2.3.0
    • None
    • Spark Core
    • None

    Description

      In practice, it was found that executor has a gc loop state, which will cause gc to spend a lot of time. Usually, this situation does not cause OOM, but causes the connection driver to timeout.

      First, we need to check the status of the gc loop,by calculating the cumulative time of GC abnormal state.By setting a reasonable threshold, save the JVM Heap before the executor heartbeat timeout

      Of course, we can also detect the unhealthy state of the driver

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              wangzhun wang-zhun
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: