Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-15658

Memory leak in S3AOutputStream

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.4
    • 3.3.0
    • fs/s3
    • None

    Description

      S3AOutputStream by calling org.apache.hadoop.fs.s3a.S3AFileSystem#createTmpFileForWrite indirectly calls java.io.File#deleteOnExit and java.io.DeleteOnExitHook which are known for memory leaking:
       
      https://bugs.java.com/view_bug.do?bug_id=6664633
      https://bugs.java.com/view_bug.do?bug_id=4872014
       
      Apparently it was even fixed (same bug but in unrelated issue) for different component couple of years ago https://issues.apache.org/jira/browse/HADOOP-8635 . 

      Attachments

        Issue Links

          Is contained by

          Sub-task - The sub-task of the issue HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              pnowojski Piotr Nowojski
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: