Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.11.0
-
None
-
None
Description
The issue occurred during a performance test of an application which reads the message send from the kafka topic, batches and compress in to avro message. Below is the snippet of code used to do the process mentioned above.
public <T> byte[] encode(T map) {
DatumWriter<T> mapWriter = new SpecificDatumWriter<>(schema);
BinaryEncoder binaryEncoder;
try (var out = new ByteArrayOutputStream())
{ binaryEncoder = EncoderFactory.get().binaryEncoder(out, null); mapWriter.write(map, binaryEncoder); binaryEncoder.flush(); return out.toByteArray(); }catch (IOException e)
{ throw new GenericException(e.getMessage(), e); }}
Exception message:
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3537) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
at org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerWrite(BufferedBinaryEncoder.java:227) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BufferedBinaryEncoder.flushBuffer(BufferedBinaryEncoder.java:96) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BufferedBinaryEncoder.ensureBounds(BufferedBinaryEncoder.java:115) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BufferedBinaryEncoder.writeInt(BufferedBinaryEncoder.java:130) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:77) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BinaryEncoder.writeString(BinaryEncoder.java:46) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.Encoder.writeString(Encoder.java:128) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:346) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:72) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:257) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:137) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[avro-1.11.0.jar!/:1.11.0]
at com.vi.edge.avro.AvroEncoder.encode(AvroEncoder.java:71) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
at com.vi.edge.avro.AvroEncoder.encodeAndCompress(AvroEncoder.java:83) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]