Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9439

KinesisReader does not report correct backlog statistics

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.22.0
    • io-java-kinesis
    • None

    Description

      The KinesisReader implementing KinesisIO reports backlog by implementing the
      UnboundedSource.getTotalBacklogBytes()
      method as opposed to the
      UnboundedSource.getSplitBacklogBytes()

      This value is supposed to represent the total backlog across all shards. This function is implemented by calling SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis shards managed within the UnboundedReader instance. As this watermark may be further ahead than the watermark across all shards, this may miss backlog bytes.

      An additional concern is that the watermark is calculated using a WatermarkPolicy, which means that the watermark may be inconsistent to the kinesis timestamp for querying backlog.

      Attachments

        Issue Links

          Activity

            People

              titmus Sebastian Graca
              scwhittle Sam Whittle
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m