Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-15796

High CPU issue in Kafka Producer when Auth Failed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.2, 3.2.3, 3.3.1, 3.3.2, 3.5.0, 3.4.1, 3.6.0, 3.5.1
    • None
    • clients, producer
    • None

    Description

      How to reproduce

      1、kafka-client 3.x.x  Producer config  enable.idempotence=true  (this is default)

      2、start kafka server , not contain client user auth info

      3、start client producer , after 3.x,producer will initProducerId and TCM state trans to INITIALIZING
      4、server reject client reqesut , producer will raise 
      AuthenticationException  (org.apache.kafka.clients.producer.internals.Sender#maybeSendAndPollTransactionalRequest)
      5、kafka-client org.apache.kafka.clients.producer.internals.Sender#runOnce catch
      AuthenticationException 
            call transactionManager.authenticationFailed(e); 
          
           synchronized void authenticationFailed(AuthenticationException e)

      {           for (TxnRequestHandler request : pendingRequests)           request.fatalError(e);       }

           this method only handle pendingRequest,but inflight request is missing 
      6、 TCM state will alway in INITIALIZING
            for judgment Condition: currentState != State.INITIALIZING && !hasProducerId()
      7、producer send mesasge , mesasge go into  batch queue,Sender will wake up and set pollTimeout=0 , prepare to send message 

      8、but , before Sender sendProducerData ,it will do message filter ,RecordAccumulator drain {}>drainBatchesForOneNode{}>shouldStopDrainBatchesForPartition 
            when producerIdAndEpoch.isValid()==false,return true, it will not collect any message 
      9、now kafka producer network thread  CPU usage will go 100%
      10、even we add user auth info and permission in kafka server ,it can not self-healing
       
       
       
      suggest : 
      also catch AuthenticationException  in  org.apache.kafka.clients.producer.internals.Sender#maybeSendAndPollTransactionalRequest  and respone failed to inflight InitProducerId request
       

      Attachments

        1. image-2023-11-07-14-18-32-016.png
          63 kB
          xiaotong.wang

        Activity

          People

            Unassigned Unassigned
            xiaotong.wang xiaotong.wang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: