Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14048

The Next Generation of the Consumer Rebalance Protocol

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Attachments

      Issue Links

        blocks

        Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14524 Modularize `core` monolith

        • Major - Major loss of function.
        • Open
        links to

        Web Link GitHub Pull Request #13203

        Web Link GitHub Pull Request #14090

        Web Link GitHub Pull Request #14327

        mentioned in

        Page Loading...

        1.
        		 Automated protocol should support namespace Sub-task Open David Jacot  
        2.
        		 Add new `group-coordinator` module Sub-task Resolved David Jacot  
        3.
        		 Introduce `GroupCoordinator` interface Sub-task Resolved David Jacot  
        4.
        		 Add ConsumerGroupHeartbeat API Sub-task Resolved David Jacot  
        5.
        		 The Kafka protocol should support nullable structs Sub-task Resolved David Jacot  
        6.
        		 New Group Coordinator State Machine Sub-task Resolved David Jacot  
        7.
        		 Implement OffsetFetch/OffsetCommit APIs Sub-task Resolved David Jacot  
        8.
        		 Implement JoinGroup/SyncGroup APIs Sub-task Resolved Jeff Kim  
        9.
        		 Implement Heartbeat API Sub-task Resolved Jeff Kim  
        10.
        		 Implement LeaveGroup API Sub-task Resolved Jeff Kim  
        11.
        		 Implement ListGroups Sub-task Resolved HaiyuanZhao  
        12.
        		 Implement DescribeGroups API Sub-task Resolved Dongnuo Lyu  
        13.
        		 Implement TnxOffsetCommit API Sub-task Resolved David Jacot  
        14.
        		 Implement DeleteGroups API and OffsetDelete API Sub-task Resolved Dongnuo Lyu  
        15.
        		 Add ConsumerGroupPrepareAssignment API Sub-task Open Unassigned  
        16.
        		 Add ConsumerGroupInstallAssignment API Sub-task Open Unassigned  
        17.
        		 Add ConsumerGroupDescribe API Sub-task In Progress Max Riedel  
        18.
        		 Extend DescribeConfigs API to support group configs Sub-task Open Jimmy Wang  
        19.
        		 Extend AlterIncrementalConfigs API to support group config Sub-task In Progress Lan Ding  
        20.
        		 Add broker side PartitionAssignor interface Sub-task Resolved David Jacot  
        21.
        		 Implement range broker side assignor Sub-task Resolved Ritika Reddy  
        22.
        		 Implement optimized uniform broker side assignor Sub-task Resolved Ritika Reddy  
        23.
        		 Implement static membeship Sub-task Resolved Sagar Rao  
        24.
        		 Implement regex subscriptions Sub-task In Progress Phuc Hong Tran  
        25.
        		 Rebalance on topic/partition metadata changes Sub-task Resolved David Jacot  
        26.
        		 Add metrics to the new coordinator Sub-task Resolved Jeff Kim  
        27.
        		 Add high watermark listener to Partition/Log layers Sub-task Resolved David Jacot  
        28.
        		 Move __consumer_offsets records from core to group-coordinator Sub-task Resolved David Jacot  
        29.
        		 Add new __consumer_offsets records Sub-task Resolved David Jacot  
        30.
        		 OffsetCommit API Version 9 Sub-task Patch Available Alexandre Dupriez  
        31.
        		 OffsetFetch API Version 9 Sub-task In Progress Alexandre Dupriez  
        32.
        		 More broker side partition assignor to common Sub-task Open David Jacot  
        33.
        		 Extend server side assignor to support rack aware replica placement Sub-task Resolved Ritika Reddy  
        34.
        		 Add ConsumerGroupHeartbeat API to AuthorizerIntegrationTest Sub-task Open Unassigned  
        35.
        		 Add support of topic id for OffsetCommitRequests in CommitRequestManager Sub-task Open Alexandre Dupriez  
        36.
        		 Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic Sub-task Open Alexandre Dupriez  
        37.
        		 Propagate Topic Ids to the Group Coordinator during Offsets Commit Sub-task Resolved Alexandre Dupriez  
        38.
        		 Propagate Topic Ids to the Group Coordinator for Offset Fetch Sub-task Open Alexandre Dupriez  
        39.
        		 Refactor ApiMessageAndVersion/Record Sub-task Open David Jacot  
        40.
        		 Validate that all partitions are assigned in TargetAssignmentBuilder Sub-task Open Unassigned  
        41.
        		 Implement Group/Offset expiration Sub-task Resolved Jeff Kim  
        42.
        		 CoordinatorPartitionWriter should reuse buffer Sub-task Resolved David Jacot  
        43.
        		 Consolidate all the group coordinator configs Sub-task Open Unassigned  
        44.
        		 Define constants for record types Sub-task Open Unassigned  
        45.
        		 Implement write operation timeout Sub-task Resolved Sagar Rao  
        46.
        		 CoordinatorContext should be protected by a lock Sub-task Resolved David Jacot  
        47.
        		 Consider replacing Subscription Metadata by a hash Sub-task Open David Jacot  
        48.
        		 Add support for OffsetCommit version 9 in admin client Sub-task Resolved Sagar Rao  
        49.
        		 Add support for OffsetCommit version 9 in consumer Sub-task Resolved Lianet Magrans  
        50.
        		 Add support for OffsetFetch version 9 in consumer Sub-task Resolved Lianet Magrans  
        51.
        		 Add support for OffsetFetch version 9 in admin Sub-task Resolved Sagar Rao  
        52.
        		 Add group type filter to ListGroups API Sub-task Resolved Ritika Reddy  
        53.
        		 Add integration test for the ListGroup API Sub-task Resolved Dongnuo Lyu  
        54.
        		 Add group type filter to the admin client Sub-task Resolved Ritika Reddy  
        55.
        		 Implement general uniform broker side assignor Sub-task Resolved Ritika Reddy  
        56.
        		 Handle heartbeat and revocation when consumer leaves group Sub-task Resolved Unassigned  
        57.
        		 Client consumer should commit offsets on close Sub-task Resolved Philip Nee  
        58.
        		 Send HB request right after reconciliation completes Sub-task Resolved Unassigned  
        59.
        		 Ensure HeartbeatRequestManager only send out some fields once Sub-task Resolved Philip Nee  
        60.
        		 Ensure coordinator node is removed upon disconnection exception Sub-task Resolved Philip Nee  
        61.
        		 Enable existing client integration tests for new protocol Sub-task Resolved Andrew Schofield  
        62.
        		 Implement client support for selecting KIP-848 server-side assignor Sub-task Resolved Lucas Brutschy  
        63.
        		 Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC Sub-task Resolved Kirk True  
        64.
        		 Implement event plumbing for ConsumerRebalanceListener callbacks Sub-task Resolved Kirk True  
        65.
        		 Implement consumer group membership state machine Sub-task Resolved Lianet Magrans  
        66.
        		 Client should stop fetching while partitions being revoked Sub-task Closed Lianet Magrans  
        67.
        		 Update client state machine to align with protocol sending one assignment at a time Sub-task Resolved Lianet Magrans  
        68.
        		 Implement auto-commit on partition assignment revocation Sub-task Closed Lianet Magrans  
        69.
        		 Update states and transitions for membership manager state machine Sub-task Resolved Lianet Magrans  
        70.
        		 Run System Tests for Old protocol in the New Coordinator Sub-task Resolved Ritika Reddy

        0%
        Original Estimate - 672h
        Remaining Estimate - 672h
        71.
        		 Member does not need to always try to join a group when a groupId is configured Sub-task Closed Philip Nee  
        72.
        		 Add integration tests for DescribeGroups API, DeleteGroups API and OffsetDelete API Sub-task Resolved Dongnuo Lyu  
        73.
        		 Add histogram metrics to GroupCoordinatorRuntimeMetrics Sub-task Open Unassigned  
        74.
        		 Refactor ConsumerRebalanceListener invocation for reuse Sub-task Resolved Kirk True  
        75.
        		 Do not send new heartbeat request while another one in-flight Sub-task Closed Philip Nee  
        76.
        		 Improve unloading logging Sub-task Resolved Ritika Reddy  
        77.
        		 Fix the different behavior in error handling between the old and new group coordinator Sub-task Resolved Dongnuo Lyu  
        78.
        		 Client support for new consumer configs Sub-task Resolved Philip Nee  
        79.
        		 New integration tests to have full coverage for preview Sub-task Closed Kirk True  
        80.
        		 Add integration tests for Heartbeat API and GroupLeave API Sub-task Resolved Dongnuo Lyu  
        81.
        		 Migrate existing integration tests to run old protocol in new coordinator Sub-task Open Dongnuo Lyu  
        82.
        		 Group Coordinator should not deliver new assignment before previous one is acknowledged Sub-task Resolved David Jacot  
        83.
        		 Implement listTopics() and partitionFor() for the AsyncKafkaConsumer Sub-task Resolved Andrew Schofield  
        84.
        		 Correct initialization of ConsumerGroupHeartbeat by client Sub-task Closed Andrew Schofield  
        85.
        		 Correct handling of KafkaConsumer.committed for new consumer Sub-task Resolved Andrew Schofield  
        86.
        		 Fix ListGroups API when runtime partition size is zero Sub-task Resolved Dongnuo Lyu  
        87.
        		 Add integration tests for JoinGroup API and SyncGroup API Sub-task Resolved Dongnuo Lyu  
        88.
        		 Move new group coordinator metrics from Yammer to Metrics Sub-task Resolved Jeff Kim  
        89.
        		 Make General Assignor's rebalance incremental Sub-task Open Unassigned  
        90.
        		 New group coordinator needs to generate snapshots while loading Sub-task Resolved Jeff Kim  
        91.
        		 Refactor polling delays Sub-task Resolved Andrew Schofield  
        92.
        		 update Group size only when groups size changes Sub-task Resolved Jeff Kim  
        93.
        		 Move GenericGroup state metrics to `GroupCoordinatorMetricsShard` Sub-task Resolved Jeff Kim  
        94.
        		 Upgrade existing generic group to consumer group Sub-task Resolved David Jacot  
        95.
        		 Ensure fairness in the uniform assignor Sub-task Resolved David Jacot  
        96.
        		 new group coordinator should check if partition goes offline during load Sub-task Resolved Jeff Kim  
        97.
        		 Add `group.coordinator.rebalance.protocols` and publish all new configs Sub-task Resolved David Jacot  
        98.
        		 Rename `Generic` to `Classic` Sub-task Resolved David Jacot  
        99.
        		 Trigger client reconciliation based on manager poll Sub-task Resolved Lucas Brutschy  
        100.
        		 Client support for OffsetFetch and OffsetCommit with topic ID Sub-task Open Lianet Magrans  
        101.
        		 Review client retry logic of OffsetFetch and OffsetCommit responses Sub-task Resolved Lianet Magrans  
        102.
        		 Client support for java regex based subscription Sub-task Resolved Phuc Hong Tran  
        103.
        		 Group commit/callbacks triggering logic Sub-task Open Lianet Magrans  
        104.
        		 Review consumer leave group request best effort and response handling Sub-task Resolved Unassigned  
        105.
        		 Review minimal effort approach on consumer last heartbeat on unsubscribe Sub-task Open Lianet Magrans  
        106.
        		 Review consumer onPartitionsAssigned called with empty partitions Sub-task Open Lianet Magrans  
        107.
        		 Topic ID integration in consumer subscription state Sub-task Open Lianet Magrans  
        108.
        		 Update list group state type filter to include the states for the new consumer group type Sub-task Resolved Lan Ding  
        109.
        		 Handle timeouts for AsyncKafkaConsumer.commitSync Sub-task Resolved Unassigned  
        110.
        		 AsyncKafkaConsumer: Always await async commit callbacks in commitSync and close Sub-task Resolved Lucas Brutschy  
        111.
        		 group size counters do not reflect the actual sizes when operations fail Sub-task Open Jeff Kim  
        112.
        		 Ensure consumer does not start fetching from added partitions until onPartitionsAssigned completes Sub-task Resolved Lianet Magrans  
        113.
        		 Review client errors thrown on OffsetFetch and OffsetCommit failures Sub-task Resolved Lianet Magrans  
        114.
        		 Add Integration test for checking if the correct assignor is chosen Sub-task Resolved Unassigned  
        115.
        		 Coordinator unloading fails when replica is deleted Sub-task Resolved David Jacot  
        116.
        		 Partition is assigned to two members at the same time Sub-task Resolved David Jacot  
        117.
        		 Implement GroupMetadataManager#onUnloaded Sub-task Resolved Jeff Kim  
        118.
        		 System test failing for new consumer on endOffsets with negative timestamps Sub-task Resolved Philip Nee  
        119.
        		 Consumer invalid transition on expired poll interval Sub-task Resolved Lianet Magrans  
        120.
        		 Implement GroupCoordinator.onPartitionsDeleted Sub-task Resolved David Jacot  
        121.
        		 Fix client reconciliation of same assignment received in different epochs Sub-task Resolved Lucas Brutschy  
        122.
        		 Extend admin to support ConsumerGroupDescribe API Sub-task Resolved David Jacot  
        123.
        		 Member should send full heartbeat when rejoining Sub-task Resolved Quoc Phong Dang  
        124.
        		 KafkaConsumer.groupMetadata() should be correct when first records are returned Sub-task Closed Bruno Cadonna  
        125.
        		 Consumer does not rejoin after fenced on delayed revocation Sub-task Resolved Lianet Magrans  
        126.
        		 Fix handling of deleted topic when auto-committing before revocation Sub-task Closed Bruno Cadonna  
        127.
        		 Console consumer fails with `IllegalStateException` Sub-task Closed Bruno Cadonna  
        128.
        		 Move code style exceptions from suppressions.xml to the code Sub-task Open David Jacot  
        129.
        		 Improve reconciliation state machine Sub-task Resolved David Jacot  
        130.
        		 Fenced member should not send heartbeats while waiting for onPartitionsLost to complete Sub-task Resolved Lianet Magrans  
        131.
        		 Stale member should trigger onPartitionsLost when leaving group Sub-task Resolved Lianet Magrans  
        132.
        		 Add group protocol migration enabling config Sub-task Resolved Dongnuo Lyu  
        133.
        		 Ensure user callbacks exceptions are propagated to the user on consumer poll Sub-task Resolved Lucas Brutschy  
        134.
        		 Classic group error responses should contain a no generation id and an empty members list Sub-task Resolved Jeff Kim  
        135.
        		 Review fenced member unsubscribe/subscribe callbacks interaction Sub-task Open Quoc Phong Dang  
        136.
        		 GroupCoordinatorService logger is not configured Sub-task Resolved Jeff Kim  
        137.
        		 fix EventAccumulator thread idle ratio metric Sub-task Resolved Jeff Kim  
        138.
        		 Offline group protocol migration Sub-task Resolved Dongnuo Lyu  
        139.
        		 Add event rate in GroupCoordinatorRuntimeMetrics Sub-task Open Jeff Kim  
        140.
        		 Offline protocol migration integration tests Sub-task Resolved Dongnuo Lyu  
        141.
        		 Full ConsumerGroupHeartbeat response must be sent when full request is received Sub-task Resolved David Jacot  
        142.
        		 High watermark updates should have a higher priority Sub-task Resolved David Jacot  
        143.
        		 Fix logic for discarding reconciliation if member rejoined Sub-task Resolved Lianet Magrans  
        144.
        		 add metric to measure time spent in purgatory Sub-task Open Jeff Kim  
        145.
        		 Online upgrade triggering and group type conversion Sub-task Resolved Dongnuo Lyu  
        146.
        		 kafka-dump-log --offsets-decoder should support new records Sub-task Resolved David Jacot  
        147.
        		 getOrMaybeCreateClassicGroup should not thrown GroupIdNotFoundException Sub-task Open Unassigned  
        148.
        		 Online downgrade triggering and group type conversion Sub-task Resolved Dongnuo Lyu  
        149.
        		 Add JMH Benchmarks for assignor performance testing Sub-task Resolved Ritika Reddy  
        150.
        		 Target Assignment Format Change Sub-task Open Ritika Reddy  
        151.
        		 Store subscription model for consumer group in group state Sub-task Resolved Ritika Reddy  
        152.
        		 JoinGroup API for upgrading ConsumerGroup Sub-task Resolved Dongnuo Lyu  
        153.
        		 Reverse Lookup Partition to Member in Assignors Sub-task Open Ritika Reddy  
        154.
        		 Uuid to String for subscribed topic names in assignment spec Sub-task Open Jeff Kim  
        155.
        		 CoordinatorRuntime write timer tasks should be cancelled once HWM advances Sub-task Resolved Jeff Kim  
        156.
        		 Re-add EventAccumulator.take(timeout) Sub-task Open Unassigned  
        157.
        		 Optimize toTopicPartitions with ConsumerProtocolSubscription Sub-task Open Dongnuo Lyu  
        158.
        		 Adjust classicGroupJoinToConsumerGroup to add subscription model Sub-task Open Dongnuo Lyu  
        159.
        		 Remove rack aware code in assignors temporarily due to performance Sub-task Resolved Ritika Reddy  
        160.
        		 SyncGroup API for upgrading ConsumerGroup Sub-task Open Dongnuo Lyu  
        161.
        		 Coalesce records into bigger batches Sub-task Open David Jacot  

        Activity

          People

            dajac David Jacot
            dajac David Jacot
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 672h
                672h
                Remaining:
                Remaining Estimate - 672h
                672h
                Logged:
                Time Spent - Not Specified
                Not Specified