Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9744

SchemaProjector fails to handle change of record namespace in Avro schema

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.3.1
    • None
    • connect
    • None

    Description

      Note: This bug report is for CP 5.3.1 / Kafka 2.3.1, but it most likely affects all versions.

      We recently changed the namespace of inner records in Avros schemas in the Confluent Schema Registry. Those changes were accepted as backwards-compatible. However, when redeploying Kafka S3 connectors consuming the relevant topics, we received error from SchemaProjector.project(), causing the connectors to crash und stop producing data:

      org.apache.kafka.connect.errors.SchemaProjectorException: Schema name mismatch. source name: my.example.Record and target name: my.example.sub.Record 

      A change of a record's namespace is compatible according to the Schema Registry (which internally uses a check from the Avro library), but not for the Connect API. I would argue that the namespace/package name should not affect compatibility, as it says nothing about the contained data and its schema. The Avro library itself does allow for differing namespaces, if the record data itself is compatible.

      Would it be possible to have a more consistent (and less restrictive) check in Connect API, so that a namespace change in the producer can be made more confidently, without fear of breaking the consuming connectors? If, on the other hand, you want to be strict about namespaces, then at least the schema registry should report a namespace change as incompatible.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              pederpansen Jurgis Pods
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: