Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8998

Avoid excessive bundle progress polling in Dataflow Runner

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Won't Fix
    • None
    • Missing
    • runner-dataflow
    • None

    Description

      Dataflow Java runner uses 0.1 secs interval for polling bundle progress from SDK Harness, and use the result to decide whether data transfer should be throttled. This can potentially overload SDK Harness. 

      We should try to come up with a way to avoid the throttling and lower the bundle progress request frequency significantly.

       

      Code reference:

      frequency setting: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java#L296

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              yichi Yichi Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: