Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7240

Kinesis IO Watermark Computation Improvements

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.14.0
    • io-java-kinesis
    • None

    Description

      Currently, watermarks in kinesis IO are computed taking into account the record arrival time in a KinesisRecord. The arrival time might not always be the right representation of the event time. The user of the IO should be able to specify how they want to extract the event time from the KinesisRecord.
      As the per current logic, the end user of the IO cannot control watermark computation in any way. A user should be able to control watermark computation through some custom heuristics or configurable params like time duration to advance the watermark if no data was received (could be due to a shard getting stalled.  The watermark should advance and not be stalled in that case).

      Attachments

        Issue Links

          Activity

            People

              ajo.thomas24 Ajo Thomas
              ajo.thomas24 Ajo Thomas
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m