Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12118

QueuingBeamFnDataClient adds polling latency to completing bundle processing

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.30.0
    • java-fn-execution
    • None

    Description

      Currently the inboundDataClients are registered with recieve, and they add data to a queue. There is no explicit indication from the clients that they are no longer going to add values to the queue.

      Within QueueingBeamFnDataClient.drainAndBlock the queue is therefore polled and if nothing is present all clients are polled to see if they are complete.

      This design makes for unfortunate tradeoffs on poll timeout:

      • cpu wasted with small timeout
      • additional latency in noticing we have completed with larger timeout

      With the existing InboundDataClient interface, we could have a separate thread call awaitCompletion on all of the clients and then shutdown the queue (adding a poison pill perhaps)
      Or we could modify InboundDataClient interface to allow registering iterest in when the client is done producing elements. The existing clients all seem based upon futures which allow that.

      Attachments

        Activity

          People

            scwhittle Sam Whittle
            scwhittle Sam Whittle
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 8h 50m
                8h 50m