Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8819

AvroCoder for SpecificRecords is not serialized correctly since 2.13.0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.13.0, 2.14.0, 2.15.0, 2.16.0
    • 2.17.0
    • runner-core
    • None

    Description

      While trying to upgrade our pipelines from Beam 2.11.0 to Beam 2.16.0 we found that our SpecificRecords used in PCollection were being decoded as GenericRecords.

      After some investigation we found the specific commit/issue that we think did brake it:

      https://github.com/apache/beam/pull/8342/files

      https://issues.apache.org/jira/browse/BEAM-7103

      After the mentioned change all AvroCoders are serialized as simple urn: "beam:coder:avro:v1" which means they are deserialized / rehydrated as AvroCoder<GenericRecord>.

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. BEAM-7103 Adding AvroGenericCoder for simple dict type cross-language data transfer

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #10218

          Web Link GitHub Pull Request #10241

          Activity

            People

              piter Piotr Szczepanik
              piter Piotr Szczepanik
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h
                  4h