Details
-
Improvement
-
Status: Resolved
-
P3
-
Resolution: Fixed
-
None
-
None
Description
The SDK worker parallelism arg is set two places, in pipeline options [1] [2] and the job server driver [3].
if pipeline.sdk_worker_parallelism > 0: pipeline.sdk_worker_parallelism is used. elif pipeline.sdk_worker_parallelism == 0: if jobServerDriver.sdkWorkerParallelism > 0: jobServerDriver.sdkWorkerParallelism is used. elif jobServerDriver.sdkWorkerParallelism == 0: the runner chooses parallelism based on cores available.
Somewhat confusingly, the default is 0 for python pipelines, but 1 for java pipelines. But anyway, jobServerDriver.sdkWorkerParallelism defaults to 1, so the comment "If 0, it will be automatically set by looking at different parameters.." is misleading, and actually only true if jobServerDriver.sdkWorkerParallelism was explicitly set to 0 as well.
Attachments
Issue Links
- is related to
-
BEAM-8387 Remove sdk-worker-parallelism option from JobServerDriver
- Resolved
- links to