Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
For streaming pipelines, the runner currently uses a single SDK harness process for all subtasks of a job that get deployed on the same task manager. In common deployments with 16 or more tasks slots, many executable stage operators all use the same SDK harness process. To scale, we need an option to run separate harness processes per subtask.
Attachments
Issue Links
- links to