Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4537

Add 'processing delay' and 'totalDelay' to the metrics reported by the Spark Streaming subsystem

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.2.1, 1.3.0
    • DStreams

    Description

      As the Spark Streaming tuning guide indicates, the key indicators of a healthy streaming job are:

      • Processing Time
      • Total Delay

      The Spark UI page for the Streaming job [1] shows these two indicators but the metrics source for Spark Streaming (StreamingSource.scala) [2] does not.

      Adding these metrics will allow external monitoring systems that consume the Spark metrics interface to track these two critical pieces of information on a streaming job performance.

      [1] https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L127

      [2] https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala

      Attachments

        Activity

          People

            Unassigned Unassigned
            gmaas Gerard Maas
            Tathagata Das Tathagata Das
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: