Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6937

Streaming FlinkTransformOverrides are not applied without explicit streaming mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.12.0
    • Component/s: runner-flink
    • Labels:
      None

      Description

      When streaming is set to false the streaming mode will be switched to true if the pipeline contains unbounded sources. There is a regression which prevents PipelineOverrides to be applied correctly in this case.

      As reported on the mailing list:

      I just upgraded to Flink 1.7.2 from 1.6.2 with my dev cluster and from Beam 2.10 to 2.11 and I am seeing this error when starting my pipelines:
      
      org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
              at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
              at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
              at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
              at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
              at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
              at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
              at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
              at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
              at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
              at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
      Caused by: java.lang.UnsupportedOperationException: The transform beam:transform:create_view:v1 is currently not supported.                                                                                                                  
              at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)                                                                                                 
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
              at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
              at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
              at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
              at org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
              at org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:68)                                                                                                                
              at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:111)                                                                                                             
              at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:108)
              at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
              at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
              at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:175)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
              ... 9 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mxm Maximilian Michels
                Reporter:
                mxm Maximilian Michels
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m