Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-16578

Revert changes to connect_distributed_test.py for the new async Consumer

    XMLWordPrintableJSON

Details

    Description

      To test the new, asynchronous Kafka Consumer implementation, we migrated a slew of system tests to run both the "old" and "new" implementations. KAFKA-16272 updated the system tests in connect_distributed_test.py so it could test the new consumer with Connect. However, we are not supporting Connect with the new consumer in 3.8.0. Unsurprisingly, when we run the Connect system tests with the new AsyncKafkaConsumer, we get errors like the following:

      test_id:    kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
      status:     FAIL
      run time:   6 minutes 3.899 seconds
      
      
          InsufficientResourcesError('Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0')
      Traceback (most recent call last):
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
          data = self.run_test()
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
          return self.test_context.function(self.test)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py", line 919, in test_exactly_once_source
          consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py", line 97, in __init__
          BackgroundThreadService.__init__(self, context, num_nodes)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py", line 26, in __init__
          super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 107, in __init__
          self.allocate_nodes()
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 217, in allocate_nodes
          self.nodes = self.cluster.alloc(self.cluster_spec)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/cluster.py", line 54, in alloc
          allocated = self.do_alloc(cluster_spec)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/finite_subcluster.py", line 31, in do_alloc
          allocated = self._available_nodes.remove_spec(cluster_spec)
        File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/node_container.py", line 117, in remove_spec
          raise InsufficientResourcesError("Not enough nodes available to allocate. " + msg)
      ducktape.cluster.node_container.InsufficientResourcesError: Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0
      

      The task here is to revert the changes made in KAFKA-16272 PR 15576.

      Attachments

        Issue Links

          Activity

            People

              kirktrue Kirk True
              kirktrue Kirk True
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: