Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5520

Flink runner per operator SDK harness option

Details

    Description

      For streaming pipelines, the runner currently uses a single SDK harness process for all subtasks of a job that get deployed on the same task manager. In common deployments with 16 or more tasks slots, many executable stage operators all use the same SDK harness process. To scale, we need an option to run separate harness processes per subtask.

         

      Attachments

        Issue Links

          Activity

            People

              thw Thomas Weise
              thw Thomas Weise
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h