Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23206 Additional Memory Tuning Metrics
  3. SPARK-24958

Add executors' process tree total memory information to heartbeat signals

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Spark Core
    • None

    Description

      Spark executors' process tree total memory information can be really useful. Currently such information are not available. The goal of this task is to compute such information for each executor, add these information to the heartbeat signals, and compute the peaks at the driver. 

      Attachments

        Issue Links

          is a parent of

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-26357 Expose executors' procfs metrics to Metrics system

          • Major - Major loss of function.
          • In Progress
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21157 Report Total Memory Used by Spark Executors

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #21916 [Github] Pull Request #21916 (rezasafi)

          Pull request #22612 [Github] Pull Request #22612 (rezasafi)

          Pull request #22612 [Github] Pull Request #22612 (rezasafi)

          Web Link GitHub Pull Request #22612

          Activity

            People

              rezasafi Reza Safi
              rezasafi Reza Safi
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: