Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
  3. HADOOP-18996

S3A to provide full support for S3 Express One Zone

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      Hadoop S3A connector has explicit awareness of and support for S3Express storage.A filesystem can now be probed for inconsistent directoriy listings through fs.hasPathCapability(path, "fs.capability.directory.listing.inconsistent"). If true, then treewalking code SHOULD NOT report a failure if, when walking into a subdirectory, a list/getFileStatus on that directory raises a FileNotFoundException.
      Show
      Hadoop S3A connector has explicit awareness of and support for S3Express storage.A filesystem can now be probed for inconsistent directoriy listings through fs.hasPathCapability(path, "fs.capability.directory.listing.inconsistent"). If true, then treewalking code SHOULD NOT report a failure if, when walking into a subdirectory, a list/getFileStatus on that directory raises a FileNotFoundException.

    Description

      HADOOP-18995 upgrades the SDK version which allows connecting to a s3 express one zone support. 

      Complete support needs to be added to address tests that fail with s3 express one zone, additional tests, documentation etc. 

      • hadoop-common path capability to indicate that treewalking may encounter missing dirs
      • use this in treewalking code in shell, mapreduce FileInputFormat etc to not fail during treewalks
      • extra path capability for s3express too.
      • tests for this
      • anything else

      A filesystem can now be probed for inconsistent directory listings through fs.hasPathCapability(path, "fs.capability.directory.listing.inconsistent")

      If true, then treewalking code SHOULD NOT report a failure if, when walking into a subdirectory, a list/getFileStatus on that directory raises a FileNotFoundExceptin.

      Attachments

        Issue Links

          contains

          Sub-task - The sub-task of the issue HADOOP-18961 S3A: add s3guard command "bucket" to create buckets

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-18955 AWS SDK v2: add path capability probe "fs.s3a.capability.aws.v2"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is a child of

          Sub-task - The sub-task of the issue HADOOP-18995 S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue SPARK-47008 Spark to support S3 Express One Zone Storage

          • Major - Major loss of function.
          • Open
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HDDS-9815 Support S3 OneZone / S3 Express API changes for Ozone

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #6308

          Activity

            People

              stevel@apache.org Steve Loughran
              ahmar Ahmar Suhail
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: