Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2889

Flink runs portable pipelines

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.10.0
    • runner-flink

    Description

      Flink should run pipelines using the full portability API as currently defined:

      https://s.apache.org/beam-fn-api
      https://s.apache.org/beam-runner-api
      https://s.apache.org/beam-job-api
      https://s.apache.org/beam-fn-api-container-contract

      This issue tracks its adoption of the portability framework. New Fn API and other features will be tracked separately.

      Attachments

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4778 Less wasteful ArtifactStagingService

          • P3 - Non-urgent bugs, features, and improvements
          • Open
          depends upon

          Sub-task - The sub-task of the issue BEAM-2876 Add provision api proto

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Sub-task - The sub-task of the issue BEAM-3883 Python SDK stages artifacts when talking to job server

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4271 Executable stages allow side input coders to be set and/or queried

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4286 Pooled artifact source

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3792 Python submits portable pipelines to the Flink-served endpoint.

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3970 Java SDK harness supports window_into

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3972 Flink runner translates batch pipelines directly by proto

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3976 SdkHarnessClient is thread-safe

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4176 Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4285 Flink batch state request handler

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4523 Implement Flink batch ExecutableStage context

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-5110 Reconile Flink JVM singleton management with deployment

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4069 Empty pipeline options can be gracefully serialized/deserialized

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4146 Python SDK sets environment in portable pipelines

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4149 SDK Harness should populate worker id in RPC headers

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4231 Runner utility for Coder instantiation

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4404 Fix docker run arguments

          • P1 - Critical bug: data loss, total loss of function, or loss of testing signal due to test failures or flakiness
          • Triage Needed

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4063 Flink runner supports cluster-wide artifact deployments through the Distributed Cache

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4068 Consistent option specification between SDKs and runners by URN

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          New Feature - A new feature of the product, which has yet to be developed. BEAM-2588 Portable Flink Runner Job API

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-2897 Configurable container/process management

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-3327 Add abstractions to manage Environment Instance lifecycles.

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4147 Abstractions for artifact delivery via arbitrary storage backends

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4615 Flink job server driver wrapper

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4056 Identify Side Inputs by PTransform ID and local name

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4067 Java: FlinkPortableTestRunner: runs portably via self-started local Flink

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4071 Java: PortableRunner - p.run() via given JobService

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4130 Portable Flink runner JobService entry point in a Docker container

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-4131 Python SDK harness container image contains SDK and dependencies

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-4148 Local server api descriptors contain urls that work on Mac and Linux

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2421 Migrate Apache Beam to use impulse primitive as the only root primitive

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2795 FlinkRunner: translate using SDK-agnostic means

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-3914 'Unzip' flattens before performing fusion

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-4267 Implement a reusable library that can run an ExecutableStage with a given Environment

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2880 Artifact server proxies

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Task - A task that needs to be done. BEAM-4098 Handle WindowInto in the Java SDK Harness

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. BEAM-5273 Local file system does not work as expected on Portability Framework with Docker

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-5464 Portable beam hangs while running TFX preprocessing step on a distributed cluster

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BEAM-5167 Use concurrency information from SDK Harness in Flink Portable Runner

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          New Feature - A new feature of the product, which has yet to be developed. BEAM-5187 Create a ProcessJobBundleFactory for non-dockerized SDK harness

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Test - A new unit, integration or system test. BEAM-5284 Enable Java Portable Flink PostCommit Tests to Jenkins

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Open

          Test - A new unit, integration or system test. BEAM-5283 Enable Python Portable Flink PostCommit Tests to Jenkins

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-2896 Portability milestone: wordcount runs everywhere

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Triage Needed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-4109 Support arbitrary artifact names in local file artifact staging service

          • P3 - Non-urgent bugs, features, and improvements
          • Open

          Bug - A problem which impairs or prevents the functions of the product. BEAM-3994 Use typed sinks and sources for FnApiControlClientPoolService

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              herohde Henning Rohde
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 13h
                  13h