Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
2.0.0
-
I am using Flink 1.2.1 running on Docker, with Task Managers distributed across different VMs as part of a Docker Swarm.
-
Patch, Important
Description
When uploading an Apache Beam application using the Flink Web UI, the parallelism set at job submission doesn't get picked up. The same happens when submitting a job using the Flink CLI.
In both cases, the parallelism ends up defaulting to 1.
When I set the parallelism programmatically within the Apache Beam code, it works: flinkPipelineOptions.setParallelism(4);
I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values passed to Flink, then defaults to 1 if it doesn't find anything.
Any ideas on how this could be fixed or worked around? I need to be able to change the parallelism dynamically, so the programmatic approach won't really work for me, nor will setting the Flink configuration at system level.
Attachments
Issue Links
- is a clone of
-
BEAM-3089 Issue with setting the parallelism at client level using Flink runner
- Resolved