Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9476

KinesisIO DescribeStream transient errors are not retried

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.21.0
    • io-java-kinesis
    • None

    Description

      During Kinesis stream setup DescribeStream is used in listShards. This API call has quota limits that can become problematic when attempting to configure multiple Kinesis streams in the same AWS account. AWS currently limits this call by default to 10 times per second for a given account. With multiple streams, this can be hit and results in a RuntimeException being thrown immediately upon the first error.

      Ideally the rate limited call can be retried a number of times before giving up instead of failing immediately.

      I have a work in progress PR that resolves this issue at https://github.com/apache/beam/pull/10973.

      Attachments

        Issue Links

          Activity

            People

              aromanenko Alexey Romanenko
              ameihm Aaron Meihm
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5h
                  5h