Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22716

Reading to ByteBuffer is broken in ParquetFooterInputFromCache

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-alpha-1
    • llap
    • None

    Description

      The ParquetFooterInputFromCache.read(ByteBuffer bb) calls the readInternal method with the result parameter passed as 'len'. The value of the result parameter will always be -1 at this point, and because of this, the readInternal method won't read anything.

        public int read(ByteBuffer bb) throws IOException {
          // Simple implementation for now - currently Parquet uses heap buffers.
          int result = -1;
          if (bb.hasArray()) {
            result = readInternal(bb.array(), bb.arrayOffset(), result);  // The readInternal is called with result=-1
            if (result > 0) {
              bb.position(bb.position() + result);
            }
          } else {
            byte[] b = new byte[bb.remaining()];
            result = readInternal(b, 0, result);     // The readInternal is called with result=-1
            bb.put(b, 0, result);
          }
          return result;
        }
      
        public int readInternal(byte[] b, int offset, int len) {
          if (position >= length) return -1;
          int argPos = offset, argEnd = offset + len;      // Here argEnd will be -1
          while (argPos < argEnd) {             // This condition will never be true, since argEnd=-1
            if (bufferIx == cacheData.length) return (argPos - offset);
            ByteBuffer data = cacheData[bufferIx].getByteBufferDup();
            int toConsume = Math.min(argEnd - argPos, data.remaining() - bufferPos);
            data.position(data.position() + bufferPos);
            data.get(b, argPos, toConsume);
            if (data.remaining() == 0) {
              ++bufferIx;
              bufferPos = 0;
            } else {
              bufferPos += toConsume;
            }
            argPos += toConsume;
          }
          return len;
        }
      

      The read(ByteBuffer bb) method wasn't called before, but in the 1.11.0 Parquet version, there were some optimizations (PARQUET-1542), so this method is called now. Because of this bug, the TestMiniLlapCliDriver and TestMiniLlapLocalCliDriver q tests are failing with the new Parquet version.

      Attachments

        1. HIVE-22716.4.patch
          1 kB
          Marta Kuczora
        2. HIVE-22716.3.patch
          1 kB
          Marta Kuczora
        3. HIVE-22716.2.patch
          1 kB
          Marta Kuczora
        4. HIVE-22716.1.patch
          1 kB
          Marta Kuczora

        Activity

          People

            kuczoram Marta Kuczora
            kuczoram Marta Kuczora
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: