Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7415

Flink runner fails to run a pipeline with TestStream source.

Details

    • Bug
    • Status: Resolved
    • P3
    • Resolution: Not A Bug
    • 2.12.0
    • Not applicable
    • runner-flink
    • None

    Description

      When I run a pipeline with the following code (on a Flink cluster):

              FlinkPipelineOptions pipelineOptions =
                      PipelineOptionsFactory.create().as(FlinkPipelineOptions.class);
              pipelineOptions.setJobName(jobName);
              pipelineOptions.setRunner(FlinkRunner.class);
              pipelineOptions.setParallelism(1);
              pipelineOptions.setStreaming(false);
              Pipeline p = Pipeline.create(pipelineOptions);
      
              p.apply(TestStream.create(AvroCoder.of(String.class)).addElements(
                  "To be, or not to be: that is the question:",
                  "Whether 'tis nobler in the mind to suffer", "The slings and arrows of fortune,",
                  "Or to take arms against a sea of troubles,").advanceWatermarkToInfinity())
      
                      .apply(ParDo.of(new DoFn<String, String>() {
      
      

      I get this error:

      org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
              at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
              at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
              at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
              at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
              at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
              at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
              at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
              at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
              at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
              at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
      Caused by: java.lang.UnsupportedOperationException: The transform urn:beam:transform:teststream:v1 is currently not supported.
              at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
              at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
              at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
              at org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
              at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:68)
              at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:111)
              at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:108)
              at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
              at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
              at flink.test.App.main(App.java:72)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
              ... 9 more
      

      The documentation doesn't specify runners that do not support this transform.

      Attachments

        Activity

          People

            Unassigned Unassigned
            srfrnk Shahar Frank
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: