Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11425 Cell/DBB end-to-end on the read-path
  3. HBASE-15077

Support OffheapKV write in compaction with out copying data on heap

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • regionserver, Scanners
    • None
    • Reviewed

    Description

      HBASE-14832 is not enough to handle this. Doing the remaining needed here.

       if (cell instanceof ByteBufferedCell) {
      890	      out.writeShort(rowLen);
      891	      ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell) cell).getRowByteBuffer(),
      892	        ((ByteBufferedCell) cell).getRowPosition(), rowLen);
      893	      out.writeByte(fLen);
      894	      ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell) cell).getFamilyByteBuffer(),
      895	        ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
      896	      ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell) cell).getQualifierByteBuffer(),
      897	        ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
      

      We have done this but it is not really helping us!
      In ByteBufferUtils#copyBufferToStream

      public static void copyBufferToStream(OutputStream out, ByteBuffer in,
            int offset, int length) throws IOException {
          if (in.hasArray()) {
            out.write(in.array(), in.arrayOffset() + offset,
                length);
          } else {
            for (int i = 0; i < length; ++i) {
              out.write(toByte(in, offset + i));
            }
          }
        }
        

      So for DBB it is so costly op writing byte by byte reading each to on heap.
      Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset, int length), it won't help us as the underlying stream is a ByteArrayOutputStream and so we will end up in copying.

      Attachments

        1. HBASE-15077.patch
          21 kB
          Anoop Sam John
        2. HBASE-15077_V2.patch
          21 kB
          Anoop Sam John
        3. HBASE-15077_V2.patch
          21 kB
          Anoop Sam John
        4. HBASE-15077_V3.patch
          21 kB
          Anoop Sam John

        Issue Links

          Activity

            People

              anoop.hbase Anoop Sam John
              anoop.hbase Anoop Sam John
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: