Details
-
Improvement
-
Status: Resolved
-
P3
-
Resolution: Duplicate
-
2.0.0
-
None
Description
KafkaIO takes parameters at construction time:
- When partitions are provided, KafkaIO actually does nothing particular at construction time.
- When only a topic is provided, I believe KafkaIO does fetch the list of partitions at construction time.
Using ValueProvider for the brokers would be a good improvement. In particular, when partitions are provided, a RuntimeProvider could be passed in.
Being able to always use a RuntimeProvider would be even better, such as when using TextIO.xxx.withoutValidation(): this would considerably improve the workflow of using Dataflow templates with KafkaIO, which currently requires to hardcode brokers into the template.
Attachments
Issue Links
- is duplicated by
-
BEAM-3925 Support ValueProviders in KafkaIO
- Open