Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-2298

Add missing futures for ConsumeKafka

    XMLWordPrintableJSON

Details

    Description

      The new ConsumeKafka processor is missing some capabilities that were present in old getKafka processor.
      1. New ConsumeKafka is not writing critical Kafka attributes i.e., kafka.key, kafka.offset, kafka.partition etc into flowFile attributes.

      Old getKafka processor:

      Standard FlowFile Attributes
      Key: 'entryDate'
      Value: 'Sun Jul 17 15:17:00 CDT 2016'
      Key: 'lineageStartDate'
      Value: 'Sun Jul 17 15:17:00 CDT 2016'
      Key: 'fileSize'
      Value: '183'
      FlowFile Attribute Map Content
      Key: 'filename'
      Value: '19709945781167274'
      Key: 'kafka.key'
      Value: '{"database":"test","table":"sc_job","pk.systemid":1}'
      Key: 'kafka.offset'
      Value: '1184010261'
      Key: 'kafka.partition'
      Value: '0'
      Key: 'kafka.topic'
      Value: ‘data'
      Key: 'path'
      Value: './'
      Key: 'uuid'
      Value: '244059bb-9ad9-4d74-b1fb-312eee72124a'

      New ConsumeKafka processor :

      Standard FlowFile Attributes
      Key: 'entryDate'
      Value: 'Sun Jul 17 15:18:41 CDT 2016'
      Key: 'lineageStartDate'
      Value: 'Sun Jul 17 15:18:41 CDT 2016'
      Key: 'fileSize'
      Value: '183'
      FlowFile Attribute Map Content
      Key: 'filename'
      Value: '19710046870478139'
      Key: 'path'
      Value: './'
      Key: 'uuid'
      Value: '349fbeb3-e342-4533-be4c-424793fa5c59’

      2. getKafka/petKafka are compatible with Kafka 0.8.x and 0.9.x .
      Please make new PublishKafka/ConsumeKafka processors based on Kafka 0.10 version.

      3. Support subscribing to multiple topics i.e., topic: topic1,topic2

      4. Support configurable Serializer/DeSerializer for String, JSON , Avro etc.

      Attachments

        Issue Links

          Activity

            People

              ozhurakousky Oleg Zhurakousky
              xmlking sumanth chinthagunta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: