Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
During Kinesis stream setup DescribeStream is used in listShards. This API call has quota limits that can become problematic when attempting to configure multiple Kinesis streams in the same AWS account. AWS currently limits this call by default to 10 times per second for a given account. With multiple streams, this can be hit and results in a RuntimeException being thrown immediately upon the first error.
Ideally the rate limited call can be retried a number of times before giving up instead of failing immediately.
I have a work in progress PR that resolves this issue at https://github.com/apache/beam/pull/10973.
Attachments
Issue Links
- links to