[SPARK-29677] Upgrade Kinesis Client - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 2.4.4
Fix Version/s: 2.4.5, 3.0.0
Component/s: DStreams
Labels:
None

Description

The current amazon-kinesis-client version is 1.8.10. This version depends on the use of `describeStream`, which has a hard limit on an AWS account (10 reqs / second). Versions 1.9.0 and up leverage `listShards`, which has no such limit. For large customers, this can be a major problem.

Upgrading the amazon-kinesis-client version should resolve this issue.

Attachments

Issue Links

links to

GitHub Pull Request #26333

Activity

People

Assignee:: Eric S Meisel

Reporter:: Eric S Meisel

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 31/Oct/19 01:07

Updated:: 16/Nov/19 00:54

Resolved:: 02/Nov/19 21:44