Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-31879

org.apache.avro.util.Utf8 cannot be serialized with avro when used in state

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      Scenario:

      Write a flink app that reads avro messages from a kafka topic.

      The avro pojos are generated with org.apache.avro.util.Utf8 type instead of java.lang.String

      When this happens, Flink logs an error message as follows:

      Class class org.apache.avro.util.Utf8 cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance. 

       

      This is problematic because `Utf8` is designed to be a fast serialized/deserialized type for Avro. But since it is not inheriting from SpecificRecordBase, it seems as if it gets handled by Kryo serializer.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ferozedaud Feroze Daud
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: