Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8121

Messages are not distributed per machines when consuming from Kafka topic with 1 partition

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Not A Problem
    • 2.14.0
    • Not applicable
    • io-java-kafka
    • None

    Description

      Messages are consumed from Kafka using KafkaIO. Each kafka topic contains only 1 partition. (That means that messages can be consumed only by one Consumer per 1 consumer group)

      When backlog of topic grows and system scales from 1 to X machines, all the messages seems to be executed onĀ  the same machine on which they are read. Due to that message throughput doesn't increase comparing X machines to 1 machine. If one machine was reading 2K messagesĀ  per s, X machines will be reading the same amount.

      Attachments

        Activity

          People

            aromanenko Alexey Romanenko
            TauJan TJ
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: