Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29312

Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      Even rows are evicted from state, we don't increase the value of "updated" metrics. The rule applies on streaming aggregations as well as streaming join, but FlatMapGroupsWithState increases the value of "updated" metrics, allowing the metrics to be shown like (total = 0, updated = 1) which looks weird for who are familiar with metrics with others.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kabhwan Jungtaek Lim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: