Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-16508

Infinte loop if output topic does not exisit

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • streams
    • None

    Description

      Kafka Streams supports `ProductionExceptionHandler` to drop records on error when writing into an output topic.

      However, if the output topic does not exist, the corresponding error cannot be skipped over because the handler is not called.

      The issue is, that the producer internally retires to fetch the output topic metadata until it times out, an a `TimeoutException` (which is a `RetriableException`) is returned via the registered `Callback`. However, for `RetriableException` there is different code path and the `ProductionExceptionHandler` is not called.

      In general, Kafka Streams correctly tries to handle as many errors a possible internally, and a `RetriableError` falls into this category (and thus there is no need to call the handler). However, for this particular case, just retrying does not solve the issue – it's unclear if throwing a retryable `TimeoutException` is actually the right thing to do for the Producer? Also not sure what the right way to address this ticket would be (currently, we cannot really detect this case, except if we would do some nasty error message String comparison what sounds hacky...)

      Attachments

        Activity

          People

            Unassigned Unassigned
            mjsax Matthias J. Sax
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: