Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17979

Interface EtagSource to allow FileStatus subclasses to provide etags

    XMLWordPrintableJSON

Details

    Description

      Various objects stores provide etags in their FileStatus implementations

      Make these values accessible

      • new interface EtagFromFileStatus to be implemented when provided
      • filesystem.md to declare requirements of etags (constant between LIST and HEAD)...
      • path capabilities for (a) etag and (b) etags consistent across rename

      Add implementation for abfs, later s3a (and google gcs)

      This is initially to handle recovery from certain failures in job commit against abfs, but it would allow a cloud-ready version of distcp to track etags of uploaded files, so diff properly.

      Attachments

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS

          • Major - Major loss of function.
          • Resolved
          causes

          Sub-task - The sub-task of the issue HADOOP-18075 ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException

          • Major - Major loss of function.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-17492 abfs listLocatedStatus to support incremental/async page fetching

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-18012 ABFS: Enable config controlled ETag check for Rename idempotency

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18425 [ABFS]: RenameFilePath Source File Not Found (404) error in retry loop

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-17981 Support etag-assisted renames in FileOutputCommitter

          • Major - Major loss of function.
          • Resolved

          Wish - General wishlist item. HADOOP-18672 ask: abfs connector to support checksum

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18002 abfs rename idempotency broken -remove recovery

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #3633

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h 10m
                  3h 10m