Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6285

add parameters for offsetConsumer in KafkaIO.read()

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.11.0
    • io-java-kafka
    • None

    Description

      Add an option in KafkaIO.read() to avoid using `offsetConsumer`. 

      What's the problem:

      When security is enabled, function `KafkaUnboundedReader.updateLatestOffsets()` always fails with lots of WARN log.

      What's the cause:

      `offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it doesn't work as expected when security is enabled. In our case, `ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer is approved.

      What's the solution:

      Add an option to disable `offsetConsumer` by developers;

      What's the side effect:

      With `offsetConsumer` disabled, backlog is unknown.

       

      rangadi any comments?

       

       

      Attachments

        Issue Links

          Activity

            People

              mingmxu Mingmin Xu
              mingmxu Mingmin Xu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5h
                  5h