Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13204 Über-jira: S3a phase III: scale and tuning
  3. HADOOP-14255

S3A to delete unnecessary fake directory objects in mkdirs()

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha4
    • fs/s3
    • None

    Description

      In S3AFileSystem, as an optimization, we delete unnecessary fake directory objects if that directory contains at least one (nested) file. That is done in closing stream of newly created file. However, if the directory becomes non-empty after we just create an empty subdirectory, we do not delete its fake directory object though that fake directory object becomes "unnecessary".

      So in S3AFileSystem::mkdirs(), we have a pending TODO:

      // TODO: If we have created an empty file at /foo/bar and we then call
      // mkdirs for /foo/bar/baz/roo what happens to the empty file /foo/bar/?
      private boolean innerMkdirs(Path p, FsPermission permission)

      This JIRA is to fix the TODO: provide consistent behavior for a fake directory object between its nested subdirectory and nested file by deleting it.

      See related discussion in HADOOP-14236. Thanks stevel@apache.org for discussion.

      Attachments

        1. HADOOP-14255.000.patch
          3 kB
          Mingliang Liu
        2. HADOOP-14255.001.patch
          4 kB
          Mingliang Liu

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14428 s3a: mkdir appears to be broken

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue HADOOP-13222 s3a.mkdirs() to delete empty fake parent directories

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-14236 S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-14594 ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-13230 S3A to optionally retain directory markers

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              liuml07 Mingliang Liu
              liuml07 Mingliang Liu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: