Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-16285

Make group metadata available when a new assignment is set in async Kafka consumer

    XMLWordPrintableJSON

Details

    Description

      Currently, the new async Kafka consumer sends an event from the background thread to the application thread when the group metadata is updated. Group metadata is updated when the background thread receives a new assignment. More specifically, the member epoch is updated each time a new assignment is received and and the member ID is updated with the first assignment.
      In contrast to the group metadata update, the assignment is directly set in the subscription without sending an update event from the background thread to the application thread. That means that there is a delay between the application thread being aware of the update to the assignment and the application thread being aware of the update to the group metadata. This behavior differs with respect to the legacy consumer were the assignment and the group metadata is updated at the same time.
      We should make the update to the group metadata available to the application thread when the update to the assignment is made available to the application thread so that assignment an group metadata are in sync.

      For example, producer.sendOffsetsToTransaction(offsetsToCommit, groupMetadata); benefits from this improvement because if the offsets to commit are consistent with the current assignment also the group metadata would be. Currently, that is not guaranteed.

      Attachments

        Issue Links

          Activity

            People

              cadonna Bruno Cadonna
              cadonna Bruno Cadonna
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: