Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-14157

Don't send requests on a closed windmill Grpc streams

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.39.0
    • runner-dataflow
    • None

    Description

      GrpcWindmillServer could send requests requests on client closed streams. This leads to windmill streams getting stalling occasionally for few seconds to few minutes. grpc-java doc says not to call onNext to send after a stream is client closed. 

      When the streams get stalled it is logged as "Output channel stalled for {}s, outbound thread {}."  from here

      Ref:

      https://github.com/grpc/grpc-java/blob/master/stub/src/main/java/io/grpc/stub/StreamObserver.java#L62

      https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java#L939

      Attachments

        Activity

          People

            arunpandianp Arun Pandian
            arunpandianp Arun Pandian
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 50m
                4h 50m