Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-19092 ABFS phase 4: post Hadoop 3.4.0 features
  3. HADOOP-18873

ABFS: AbfsOutputStream doesnt close DataBlocks object.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.4
    • 3.3.4
    • None

    Description

      AbfsOutputStream doesnt close the dataBlock object created for the upload.

      What is the implication of not doing that:
      DataBlocks has three implementations:

      1. ByteArrayBlock
        1. This creates an object of DataBlockByteArrayOutputStream (child of ByteArrayOutputStream: wrapper arround byte-arrray for populating, reading the array.
        2. This gets GCed.
      2. ByteBufferBlock:
        1. There is a defined DirectBufferPool from which it tries to request the directBuffer.
        2. If nothing in the pool, a new directBuffer is created.
        3. the `close` method on the this object has the responsiblity of returning back the buffer to pool so it can be reused.
        4. Since we are not calling the `close`:
          1. The pool is rendered of less use, since each request creates a new directBuffer from memory.
          2. All the object can be GCed and the direct-memory allocated may be returned on the GC. What if the process crashes, the memory never goes back and cause memory issue on the machine.
      3. DiskBlock:
        1. This creates a file on disk on which the data-to-upload is written. This file gets deleted in startUpload().close().

       

      startUpload() gives an object of BlockUploadData which gives method of `toByteArray()` which is used in abfsOutputStream to get the byteArray in the dataBlock.

       

      Method which uses the DataBlock object: https://github.com/apache/hadoop/blob/fac7d26c5d7f791565cc3ab45d079e2cca725f95/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java#L298

      Attachments

        Issue Links

          Activity

            People

              pranavsaxena Pranav Saxena
              pranavsaxena Pranav Saxena
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: