Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8582

Python SDK emits duplicate records for Default and AfterWatermark triggers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Triage Needed
    • P2
    • Resolution: Fixed
    • None
    • 2.19.0
    • sdk-py-core
    • None

    Description

      This was found after fixing https://issues.apache.org/jira/browse/BEAM-8581. The fix for 8581 was to pass in the input watermark. Previously, it was using MIN_TIMESTAMP for all of its EOW calculations. By giving it a proper input watermark, this bug started to manifest.

      The DefaultTrigger and AfterWatermark do not clear their timers after the watermark passed the end of the endow, leading to duplicate records being emitted.

      Fix: Clear the watermark timer when the watermark reaches the end of the window.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. BEAM-10054 Direct Runner execution stalls with test pipeline

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Activity

            People

              rohdesam Sam Rohde
              rohdesam Sam Rohde
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: