Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14048 The Next Generation of the Consumer Rebalance Protocol
  3. KAFKA-15997

Ensure fairness in the uniform assignor

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

       

       

      Fairness has to be ensured in uniform assignor as it was in cooperative-sticky one.

      There's this test 0113 subtest u_multiple_subscription_changes in librdkafka where 8 consumers are subscribing to the same topic, and it's verifying that all of them are getting 2 partitions assigned. But with new protocol it seems two consumers get assigned 3 partitions and 1 has zero partitions. The test doesn't configure any client.rack.

      [0113_cooperative_rebalance  /478.183s] Consumer assignments (subscription_variation 0) (stabilized) (no rebalance cb):
[0113_cooperative_rebalance  /478.183s] Consumer C_0#consumer-3 assignment (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [5] (2000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [8] (4000msgs)
[0113_cooperative_rebalance  /478.183s] Consumer C_1#consumer-4 assignment (3): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [0] (1000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [3] (2000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [13] (1000msgs)
[0113_cooperative_rebalance  /478.184s] Consumer C_2#consumer-5 assignment (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [6] (1000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [10] (2000msgs)
[0113_cooperative_rebalance  /478.184s] Consumer C_3#consumer-6 assignment (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [7] (1000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [9] (2000msgs)
[0113_cooperative_rebalance  /478.184s] Consumer C_4#consumer-7 assignment (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [11] (1000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [14] (3000msgs)
[0113_cooperative_rebalance  /478.184s] Consumer C_5#consumer-8 assignment (3): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [1] (2000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [2] (2000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [4] (1000msgs)
[0113_cooperative_rebalance  /478.184s] Consumer C_6#consumer-9 assignment (0): 
[0113_cooperative_rebalance  /478.184s] Consumer C_7#consumer-10 assignment (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [12] (1000msgs), rdkafkatest_rnd24419cc75e59d8de_0113u_1 [15] (1000msgs)
[0113_cooperative_rebalance  /478.184s] 16/32 partitions assigned
[0113_cooperative_rebalance  /478.184s] Consumer C_0#consumer-3 has 2 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_1#consumer-4 has 3 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_2#consumer-5 has 2 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_3#consumer-6 has 2 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_4#consumer-7 has 2 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_5#consumer-8 has 3 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_6#consumer-9 has 0 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[0113_cooperative_rebalance  /478.184s] Consumer C_7#consumer-10 has 2 assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
[<MAIN>                      /479.057s] 1 test(s) running: 0113_cooperative_rebalance
[<MAIN>                      /480.057s] 1 test(s) running: 0113_cooperative_rebalance
[<MAIN>                      /481.057s] 1 test(s) running: 0113_cooperative_rebalance
[0113_cooperative_rebalance  /482.498s] TEST FAILURE
### Test "0113_cooperative_rebalance (u_multiple_subscription_changes:2390: use_rebalance_cb: 0, subscription_variation: 0)" failed at test.c:1243:check_test_timeouts() at Thu Dec  7 15:52:15 2023: ###
Test 0113_cooperative_rebalance (u_multiple_subscription_changes:2390: use_rebalance_cb: 0, subscription_variation: 0) timed out (timeout set to 480 seconds)
./run-test.sh: line 62: 3512920 Killed                  $TEST $ARGS
###
### Test ./test-runner in bare mode FAILED! (return code 137) ###
###

       

       

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue KAFKA-16249 Improve reconciliation state machine

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              dajac David Jacot
              emasab Emanuele Sabellico
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: