Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features
  3. HADOOP-16756

distcp -update to S3A; abfs, etc always overwrites due to block size mismatch

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.0.4, 3.2.2, 3.3.1
    • fs/s3, tools/distcp
    • None

    Description

      Distcp over S3A always copies all source files no matter the files are changed or not. This is opposite to the statement in the doc below.

      http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

      And to use -update to only copy changed files.

      CopyMapper compares file length as well as block size before copying. While the file length should match, the block size does not. This is apparently because the returned block size from S3A is always 32MB.

      https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348

      I'd suppose we should update the documentation or make code change.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-17256 DistCp -update option will be invalid when distcp files from hdfs to S3

          • Major - Major loss of function.
          • Resolved
          is broken by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8143 Change distcp to have -pb on by default

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-15300 distcp -update to WASB and ADL copies up all the files, always

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16932 distcp copy calls getFileStatus() needlessly and can fail against S3

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              daisuke.kobayashi Daisuke Kobayashi
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: