Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5538

Create a ConvertProtobufToJson procesor

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Extensions
    • Labels:
      None

      Description

      Protocol Buffers (protobuf) is a serialization format: https://developers.google.com/protocol-buffers/

      It is somewhat similar to AVRO, but the schema is not normally carried alongside the data, so messages are not self-describing. Instead, a separate .proto schema definition is written and the protoc binary tool used to generate bindings for the appropriate language (Java, Python, C++ etc).

      It would be great to be able to handle protobuf in NiFi. Full support as a record reader/writer would be ideal, but is a fair amount of work and when I (briefly) looked at it, the use of a separate schema definition language looked tricky.

      Pre-empting the question why not use https://github.com/whiver/nifi-protobuf-processor, that includes the capability to compile a .proto schema definition, which includes C++ binary executables inside the nar. For my use case that is not permitted (but I fully accept others may find this k). Instead, my processor requires the .proto schema to be compiled using the protoc tool to a Field Descriptor Set, which is a compiled protobuf encoded blob which describes the schema (see https://developers.google.com/protocol-buffers/docs/techniques#self-description).

      For my use case, conversion to JSON is adequate and something I have already implemented. If there is enough community interest, I will go through my organisation's release process (about as much effort as implementing and testing the processor, and much less fun!).

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jrs53 James Srinivasan
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: