Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14336

Reduce allocations from MetadataResponse#convertToNodeArray

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • admin
    • None

    Description

      While profiling a Kafka consumer application that utilizes org.apache.kafka.clients.admin.Admin#listOffsets(java.util.Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.admin.OffsetSpec>), one of the largest aggregate memory allocations are coming from org.apache.kafka.common.requests.MetadataResponse.convertToNodeArray(List, Map) due to the use of stream reference pipeline.

      We can avoid allocating the stream reference pipeline & spliterator for this case by explicitly allocating the presized Node[] and using a for loop with int induction over the specified IDs List<Integer> argument.

      java.util.stream.ReferencePipeline$3
at java.util.stream.ReferencePipeline.map(Function)
at org.apache.kafka.common.requests.MetadataResponse.convertToNodeArray(List, Map)
   at org.apache.kafka.common.requests.MetadataResponse.toPartitionInfo(MetadataResponse$PartitionMetadata, Map)
   at org.apache.kafka.common.requests.MetadataResponse.cluster()
   at org.apache.kafka.clients.admin.KafkaAdminClient.getListOffsetsCalls(MetadataOperationContext, Map, Map)
   at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$listOffsets$21(MetadataOperationContext, Map, Map)
   at org.apache.kafka.clients.admin.KafkaAdminClient$$Lambda$3539+0x000000080142d120.1364252822.get()
   at org.apache.kafka.clients.admin.KafkaAdminClient$23.handleResponse(AbstractResponse)
   at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(long, List)
   at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests()
   at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run()
   at java.lang.Thread.run()

      JFR Snippet:

      Attachments

        1. jfr-snippet.png
          864 kB
          David Schlosnagle

        Issue Links

          links to

          Web Link GitHub Pull Request #12782

          Activity

            People

              Unassigned Unassigned
              schlosna David Schlosnagle
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: