Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9573

Watermark hold for timer output timestamp is not computed correctly

Details

    • Bug
    • Status: Resolved
    • P0
    • Resolution: Fixed
    • 2.20.0
    • 2.20.0
    • runner-flink
    • None

    Description

      With the introduction of timer output timestamp, a new watermark hold had been added to the Flink Runner. The watermark computation works on the keyed state backend which computes a key-scoped watermark hold and not the desired operator-wide watermark hold.

      Computation: https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140

      Key-scoped state: https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130

      We need to change this to operate on all keys. This has to be done before fixing BEAM-9566.

      Attachments

        Issue Links

          Activity

            People

              mxm Maximilian Michels
              mxm Maximilian Michels
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8.5h
                  8.5h