Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8382

Add polling interval to KinesisIO.Read

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.13.0, 2.14.0, 2.15.0
    • 2.21.0
    • io-java-kinesis
    • None

    Description

      With the current implementation we are observing Kinesis throttling due to ReadProvisionedThroughputExceeded on the order of hundreds of times per second, regardless of the actual Kinesis throughput. This is because the ShardReadersPool readLoop() method is polling getRecords() as fast as possible.

      From the KDS documentation:

      Each shard can support up to five read transactions per second.

      and

      For best results, sleep for at least 1 second (1,000 milliseconds) between calls to getRecords to avoid exceeding the limit on getRecords frequency.

      https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html

      https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html

      Attachments

        Issue Links

          Activity

            People

              jfarr Jonothan Farr
              jfarr Jonothan Farr
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 15h 50m
                  15h 50m