Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11910

Increase subsequent page size for bags after the first

Details

    Description

      Currently the page size of bags requested from the streaming dataflow backend is always 8MB. In pipelines with large bags this can limit throughput as it results in more round-trips to the backend. In particular with Streaming Engine this is noticable due to increased latency.

      I propose using 8MB for the first bag fetch and then doubling the limit for subsequent paginations

      Attachments

        Issue Links

          Activity

            People

              scwhittle Sam Whittle
              scwhittle Sam Whittle
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h