Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29450

[SS] In streaming aggregation, metric for output rows is not measured in append mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.4.4, 3.0.0
    • Fix Version/s: None
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      The bug is reported in dev. mailing list. Credit to Jacek Laskowski. Quoting here:
       

      I've just noticed that the number of output rows metric of StateStoreSaveExec physical operator does not seem to be measured for Append output mode. In other words, whatever happens before or after StateStoreSaveExec operator the metric is always 0.
       
      It is measured for the other modes - Complete and Update.

      https://lists.apache.org/thread.html/eec318f7ff84c700bffc8bdb7c95c0dcaffb2494ac7ccd7a7c7c6588@%3Cdev.spark.apache.org%3E

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kabhwan Jungtaek Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: