Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Add an option in KafkaIO.read() to avoid using `offsetConsumer`.
What's the problem:
When security is enabled, function `KafkaUnboundedReader.updateLatestOffsets()` always fails with lots of WARN log.
What's the cause:
`offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it doesn't work as expected when security is enabled. In our case, `ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer is approved.
What's the solution:
Add an option to disable `offsetConsumer` by developers;
What's the side effect:
With `offsetConsumer` disabled, backlog is unknown.
rangadi any comments?
Attachments
Issue Links
- links to