Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5999

Proto error when running test_pardo_timers of Python PortableValidatesRunner

    XMLWordPrintableJSON

    Details

      Description

      When I run FlinkRunnerTest.test_pardo_timers, I get the following error during fusing the Proto pipeline. Note, the Java ValidatesPortableRunner timer tests run fine. Seems to be an issue with the Python translation.

      [flink-runner-job-server] ERROR org.apache.beam.runners.flink.FlinkJobInvocation - Error during job invocation test_pardo_timers_1541532770.41_45967eac-8089-4a5f-9d2b-7a4a9d01c7f1.
      java.lang.IllegalArgumentException
      	at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.getInputsOrThrow(RunnerApi.java:7418)
      	at org.apache.beam.runners.core.construction.graph.QueryablePipeline.lambda$getTimers$11(QueryablePipeline.java:385)
      	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      	at java.util.Iterator.forEachRemaining(Iterator.java:116)
      	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
      	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
      	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
      	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
      	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
      	at org.apache.beam.runners.core.construction.graph.QueryablePipeline.getTimers(QueryablePipeline.java:392)
      	at org.apache.beam.runners.core.construction.graph.GreedyStageFuser.forGrpcPortRead(GreedyStageFuser.java:93)
      	at org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuseSiblings(GreedyPipelineFuser.java:340)
      	at org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fusePipeline(GreedyPipelineFuser.java:142)
      	at org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.<init>(GreedyPipelineFuser.java:75)
      	at org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:88)
      	at org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:96)
      	at org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
      	at org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
      	at org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      To reproduce, enable the test in FlinkRunnerTest, then run the following from sdks/python:

      python -m apache_beam.runners.portability.flink_runner_test --flink_job_server_jar=/path/to//beam/runners/flink/job-server/build/libs/beam-runners-flink_2.11-job-server-2.9.0-SNAPSHOT.jar --streaming FlinkRunnerTest.test_pardo_timers
      

        Attachments

          Activity

            People

            • Assignee:
              robertwb Robert Bradshaw
              Reporter:
              mxm Maximilian Michels
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 40m
                4h 40m