Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
The ParquetFooterInputFromCache.read(ByteBuffer bb) calls the readInternal method with the result parameter passed as 'len'. The value of the result parameter will always be -1 at this point, and because of this, the readInternal method won't read anything.
public int read(ByteBuffer bb) throws IOException { // Simple implementation for now - currently Parquet uses heap buffers. int result = -1; if (bb.hasArray()) { result = readInternal(bb.array(), bb.arrayOffset(), result); // The readInternal is called with result=-1 if (result > 0) { bb.position(bb.position() + result); } } else { byte[] b = new byte[bb.remaining()]; result = readInternal(b, 0, result); // The readInternal is called with result=-1 bb.put(b, 0, result); } return result; }
public int readInternal(byte[] b, int offset, int len) { if (position >= length) return -1; int argPos = offset, argEnd = offset + len; // Here argEnd will be -1 while (argPos < argEnd) { // This condition will never be true, since argEnd=-1 if (bufferIx == cacheData.length) return (argPos - offset); ByteBuffer data = cacheData[bufferIx].getByteBufferDup(); int toConsume = Math.min(argEnd - argPos, data.remaining() - bufferPos); data.position(data.position() + bufferPos); data.get(b, argPos, toConsume); if (data.remaining() == 0) { ++bufferIx; bufferPos = 0; } else { bufferPos += toConsume; } argPos += toConsume; } return len; }
The read(ByteBuffer bb) method wasn't called before, but in the 1.11.0 Parquet version, there were some optimizations (PARQUET-1542), so this method is called now. Because of this bug, the TestMiniLlapCliDriver and TestMiniLlapLocalCliDriver q tests are failing with the new Parquet version.