Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8471

Flink native job submission for portable pipelines

    XMLWordPrintableJSON

    Details

      Description

      There are currently two methods to run a portable pipeline written in a non-JVM language to Flink:

      1) Run the SDK client entry point which will submit the job server, which in turn will submit to a Flink cluster using the Flink remote environment

      2) Run the SDK client entry point to generate a Flink jar file that can be used to start the Flink job using any of the Flink client tooling available.

      Either approach requires the SDK client and the job server dependency to be present on the client. This doesn't work well in environments such as FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l).

      This improvement is to provide a new Flink entry point (main method) that invokes the SDK client entry point to generate the pipeline and submits the resulting Flink job like any other Flink native driver program would, via the optimizer plan environment ("[auto]").

       

        Attachments

          Activity

            People

            • Assignee:
              thw Thomas Weise
              Reporter:
              thw Thomas Weise
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 50m
                5h 50m