Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-13600

S3a rename() to copy files in a directory in parallel

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.7.3
    • HADOOP-15183
    • fs/s3
    • None

    Description

      Currently a directory rename does a one-by-one copy, making the request O(files * data). If the copy operations were launched in parallel, the duration of the copy may be reducable to the duration of the longest copy. For a directory with many files, this will be significant

      Attachments

        1. HADOOP-13600.001.patch
          25 kB
          Sahil Takiar

        Issue Links

        depends upon

        Sub-task - The sub-task of the issue HADOOP-13823 s3a rename: fail if dest file exists

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        is related to

        Sub-task - The sub-task of the issue HIVE-15093 S3-to-S3 Renames: Files should be moved individually rather than at a directory level

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #157

        Web Link GitHub Pull Request #157

        Web Link GitHub Pull Request #167

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment