Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
Currently the page size of bags requested from the streaming dataflow backend is always 8MB. In pipelines with large bags this can limit throughput as it results in more round-trips to the backend. In particular with Streaming Engine this is noticable due to increased latency.
I propose using 8MB for the first bag fetch and then doubling the limit for subsequent paginations
Attachments
Issue Links
- links to