Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2889 Flink runs portable pipelines
  3. BEAM-3673

FlinkRunner: Harness manager for connecting operators to SDK Harnesses

Details

    • Sub-task
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.6.0
    • runner-flink
    • None

    Description

      SDK harnesses require a common set of gRPC services to operate. The role of the harness manager is to provide a uniform interface that multiplexes data streams and auxiliary data between SDK environments and operators within a given job.

      Note that multiple operators may communicate with a single SDK environment to amortize container initialization cost. Environments are not shared between different jobs.

      The initial implementation will shell out to local docker, but the harness manager should eventually support working with externally-managed environments (e.g., created by Kubernetes).

      Attachments

        Activity

          People

            axelmagn Axel Magnuson
            bsidhom Ben Sidhom
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: