Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18781

ABFS backReference passed down to streams to avoid GC closing the FS.

    XMLWordPrintableJSON

Details

    Description

      Applications using AzureBlobFileSystem to create the AbfsOutputStream can use the AbfsOutputStream for the purpose of writing, however, the OutputStream doesn't hold any reference to the fs instance that created it, which can make the FS instance eligible for GC, when this occurs, AzureblobFileSystem's `finalize()` method gets called which in turn closes the FS, and in turn call the close for AzureBlobFileSystemStore, which uses the same Threadpool that is used by the AbfsOutputStream. This leads to the closing of the thread pool while the writing is happening in the background and leads to hanging while writing.

       

      Solution:
      Pass a backreference of AzureBlobFileSystem into AzureBlobFileSystemStore and AbfsOutputStream as well.

       

      Same should be done for AbfsInputStream as well.

      Attachments

        Activity

          People

            mehakmeetSingh Mehakmeet Singh
            mehakmeetSingh Mehakmeet Singh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: