Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Not A Problem
-
2.14.0
-
None
Description
Messages are consumed from Kafka using KafkaIO. Each kafka topic contains only 1 partition. (That means that messages can be consumed only by one Consumer per 1 consumer group)
When backlog of topic grows and system scales from 1 to X machines, all the messages seems to be executed onĀ the same machine on which they are read. Due to that message throughput doesn't increase comparing X machines to 1 machine. If one machine was reading 2K messagesĀ per s, X machines will be reading the same amount.