Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40460

Streaming metrics is zero when select _metadata

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0, 3.3.1, 3.3.2, 3.4.0
    • 3.3.1, 3.4.0
    • Structured Streaming
    • None

    Description

      Streaming metrics report all 0 (`processedRowsPerSecond`, etc) when selecting `_metadata` column. Because the logical plan from the batch and the actual planned logical are mismatched: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala#L348

      Attachments

        Activity

          People

            yaohua Yaohua Zhao
            yaohua Yaohua Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: