Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-3719

OutOfMemoryError issue in BinaryEncoder - EncoderFactory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.11.0
    • None
    • java
    • None

    Description

      The issue occurred during a performance test of an application which reads the message send from the kafka topic, batches and compress in to avro message. Below is the snippet of code used to do the process mentioned above.

      public <T> byte[] encode(T map) {
          DatumWriter<T> mapWriter = new SpecificDatumWriter<>(schema);
          BinaryEncoder binaryEncoder;

          try (var out = new ByteArrayOutputStream())

      {       binaryEncoder = EncoderFactory.get().binaryEncoder(out, null);       mapWriter.write(map, binaryEncoder);       binaryEncoder.flush();       return out.toByteArray();     }

      catch (IOException e)

      {       throw new GenericException(e.getMessage(), e);     }

        }

      Exception message:

      java.lang.OutOfMemoryError: Java heap space
          at java.base/java.util.Arrays.copyOf(Arrays.java:3537) ~[na:na]
          at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
          at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
          at org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerWrite(BufferedBinaryEncoder.java:227) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.io.BufferedBinaryEncoder.flushBuffer(BufferedBinaryEncoder.java:96) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.io.BufferedBinaryEncoder.ensureBounds(BufferedBinaryEncoder.java:115) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.io.BufferedBinaryEncoder.writeInt(BufferedBinaryEncoder.java:130) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:77) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.io.BinaryEncoder.writeString(BinaryEncoder.java:46) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.io.Encoder.writeString(Encoder.java:128) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:346) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:72) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:257) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:137) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
          at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[avro-1.11.0.jar!/:1.11.0]
          at com.vi.edge.avro.AvroEncoder.encode(AvroEncoder.java:71) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
          at com.vi.edge.avro.AvroEncoder.encodeAndCompress(AvroEncoder.java:83) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]

      Attachments

        Activity

          People

            Unassigned Unassigned
            sandeep-iot Sandeep Rathinam
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: