Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6512

[beam_PreCommit_Java_Cron] [GrpcDataServiceTest] Flake, Multiplexer hanging up

Details

    Description

      Use this form to file an issue for test failure:

      Initial investigation:

      I see this message:

      org.apache.beam.runners.fnexecution.data.GrpcDataServiceTest > testMessageReceivedBySingleClientWhenThereAreMultipleClients FAILED
          java.lang.AssertionError at GrpcDataServiceTest.java:105
      

      And this message:

      [grpc-default-executor-2] WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Hanged up for unknown endpoint.
      Jan 25, 2019 6:16:02 PM org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor run
      SEVERE: Exception while executing runnable org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed@41415b8e
      java.lang.RuntimeException: org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: CANCELLED: Multiplexer hanging up
      	at org.apache.beam.sdk.fn.test.TestStreams.lambda$throwingErrorHandler$0(TestStreams.java:95)
      	at org.apache.beam.sdk.fn.test.TestStreams$ForwardingCallStreamObserver.onError(TestStreams.java:144)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: CANCELLED: Multiplexer hanging up
      	at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asRuntimeException(Status.java:526)
      	... 19 more
      

      Note that the second message is something I've seen when successfully ending a job without in portable runners (specifically the Java Reference Runner), so it might be a red herring.

      Attachments

        Issue Links

          Activity

            People

              bhulette Brian Hulette
              danoliveira Daniel Oliveira
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m