Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
In the flink portable runner, we currently support two options for sdk worker parallelism (how many python worker processes we run). The default is one per taskmanager, and with --sdk-worker-parallelism=stage you get one per stage. However, for complex pipelines with many beam operators that get fused into a single flink task this can produce hundreds of worker processes per TM.
Flink uses the notion of task slots to limit resource utilization on a box; I think that beam should try to respect those limits as well. I think ideally we'd produce a single python worker per task slot/flink operator chain.
Attachments
Issue Links
- links to