Details
-
Bug
-
Status: Resolved
-
P1
-
Resolution: Fixed
-
None
Description
The same issue has been pointed out and the issues were marked resolved. But they were still remaining parts....
https://issues.apache.org/jira/browse/BEAM-5412?jql=text%20~%20%22tfrecord%22
Issue # 1: TFRecordCodec only tries once to read the header/footer. This is likely to fail around the end of channel buffer.
Issue # 2: (minor) TFRecordCodec currently does not checks how much it writes.
Seems like it only happens with Zstd compression (or any other picky input stream that refuse to read fully). ZstdInputStream seems very picky at giving out data.
The parts with the issue are
And not so problem within the beam application (As all (or most) of WritableByteChannels in beam-java-sdk-core are backed by some OutputStream), but still not following the WritableByteChannel specification,
ReadableByteChannel/WritableByteChannel Javadoc specifies that they are not required to read/write fully, and can refuse to read/write time to time.
Attachments
Issue Links
- links to