Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8085

Use poll(Duration) in ConsumeKafka_2_x processors

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.13.0
    • Extensions
    • None

    Description

      The currently used KafkaConsumer.poll(long timeout) method is deprecated in 2.x. The method waits for the metadata response indefinitely until we get it (so in case of an unavailable/stopped kafka cluster until the connection gets recovered). It means the poll(long) acquires and holds a thread exclusively for this period which can lead to starvation of other processors for timer driven threads.

      Use poll(Duration timeout) instead which is not simply an overloaded version of the method but has different semantics (https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior).
      If can return from metadata fetch after the timeout expires and does not occupy a thread continuously and makes the processor able to yield.

      Attachments

        Issue Links

          Activity

            People

              turcsanyip Peter Turcsanyi
              turcsanyip Peter Turcsanyi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h