Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23206 Additional Memory Tuning Metrics
  3. SPARK-34898

Send ExecutorMetricsUpdate EventLog appropriately

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • Spark Core
    • None

    Description

      In current EventLoggingListener, we won't write SparkListenerExecutorMetricsUpdate message at all

      override def onExecutorMetricsUpdate(event: SparkListenerExecutorMetricsUpdate): Unit = {
        if (shouldLogStageExecutorMetrics) {
          event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
            liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor) =>
              // If the update came from the driver, stageKey1 will be the dummy key (-1, -1),
              // so record those peaks for all active stages.
              // Otherwise, record the peaks for the matching stage.
              if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
                val metrics = metricsPerExecutor.getOrElseUpdate(
                  event.execId, new ExecutorMetrics())
                metrics.compareAndUpdatePeakValues(newPeaks)
              }
            }
          }
        }
      }
      

      It causes this effect that we can't get driver peakMemoryMetrics in SHS.  We can get executor's since it will update with TaskEnd events.

      Attachments

        Activity

          People

            apachespark Apache Spark
            angerszhuuu angerszhu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: