Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2449

Support ValueProviders in KafkaIO

Details

    • Improvement
    • Status: Resolved
    • P3
    • Resolution: Duplicate
    • 2.0.0
    • Not applicable
    • io-java-kafka
    • None

    Description

      KafkaIO takes parameters at construction time:

      • When partitions are provided, KafkaIO actually does nothing particular at construction time.
      • When only a topic is provided, I believe KafkaIO does fetch the list of partitions at construction time.

      Using ValueProvider for the brokers would be a good improvement. In particular, when partitions are provided, a RuntimeProvider could be passed in.

      Being able to always use a RuntimeProvider would be even better, such as when using TextIO.xxx.withoutValidation(): this would considerably improve the workflow of using Dataflow templates with KafkaIO, which currently requires to hardcode brokers into the template.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              peay peay
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: