Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18073

S3A: Upgrade AWS SDK to V2

    XMLWordPrintableJSON

Details

    • Incompatible change
    • Hide
      The S3A connector now uses the V2 AWS SDK.
      This is a significant change at the source code level.
      Any applications using the internal extension/override points in
      the filesystem connector are likely to break.
      Consult the document aws_sdk_upgrade for the full details.
      Show
      The S3A connector now uses the V2 AWS SDK. This is a significant change at the source code level. Any applications using the internal extension/override points in the filesystem connector are likely to break. Consult the document aws_sdk_upgrade for the full details.

    Description

      This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java V1 to AWS SDK for Java V2.

      Original use case:

      We would like to access s3 with AWS SSO, which is supported in software.amazon.awssdk:sdk-core:2.*.

      In particular, from https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html, when to set 'fs.s3a.aws.credentials.provider', it must be "com.amazonaws.auth.AWSCredentialsProvider". We would like to support "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which supports AWS SSO, so users only need to authenticate once.

      Attachments

        1. Upgrading S3A to SDKV2.pdf
          204 kB
          Ahmar Suhail

        Issue Links

          breaks

          New Feature - A new feature of the product, which has yet to be developed. SPARK-38958 Override S3 Client in Spark Write/Read calls

          • Major - Major loss of function.
          • Open
          causes

          Bug - A problem which impairs or prevents the functions of the product. HBASE-28056 [HBoss] add support for AWS v2 SDK

          • Major - Major loss of function.
          • In Progress
          is a child of

          Sub-task - The sub-task of the issue HADOOP-18995 S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features

          • Major - Major loss of function.
          • Open
          is related to

          Sub-task - The sub-task of the issue HADOOP-18352 Support AWS IAM Identity Centre (prev. AWS SSO) for providing credentials to S3A

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-44124 Upgrade AWS SDK to v2

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18286 S3a: allow custom retry policies

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link GitHub Pull Request #4706

          Web Link GitHub Pull Request #4944

          Web Link GitHub Pull Request #5163

          Web Link GitHub Pull Request #5684

          Web Link GitHub Pull Request #5707

          Web Link GitHub Pull Request #5857

          Web Link GitHub Pull Request #5858

          Web Link GitHub Pull Request #5980

          Web Link GitHub Pull Request #5981

          Web Link GitHub Pull Request #6303

          1.
          		 Upgrade AWS SDK to V2 - Prerequisites Sub-task Resolved Ahmar Suhail

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          2.
          		 AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers Sub-task Resolved Ahmar Suhail  
          3.
          		 AWS SDK V2 - Complete outstanding items Sub-task Resolved Ahmar Suhail  
          4.
          		 AWS SDK V2 - Update region logic Sub-task Resolved Unassigned  
          5.
          		 AWS SDK V2 - Qualify the upgrade. Sub-task Resolved Unassigned  
          6.
          		 AWS SDK V2 - Fix failing tests Sub-task Resolved Unassigned  
          7.
          		 AWS SDK v2 code tuning Sub-task Resolved Steve Loughran  
          8.
          		 AWS v2 SDK: stabilise dependencies with rest of hadoop libraries Sub-task Resolved Steve Loughran  
          9.
          		 convert declarations of AWS v1 SDK EnvironmentVariableCredentialsProvider to v2 version Sub-task Resolved Unassigned  
          10.
          		 AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure Sub-task Resolved Steve Loughran  
          11.
          		 s3a DelegationToken plugin to expand return type of deploy/binding Sub-task Resolved Steve Loughran  
          12.
          		 LogExactlyOnce to add a debug() method Sub-task Resolved Steve Loughran  
          13.
          		 list AWS SDK v2 libraries in LICENSE-binary Sub-task Resolved Steve Loughran  
          14.
          		 Merge aws v2 upgrade feature branch into trunk Sub-task Resolved Steve Loughran  
          15.
          		 AWS SDK v2 build complaints Sub-task Resolved Steve Loughran  
          16.
          		 AWS SDK v2: make the v1 bridging support optional Sub-task Resolved Steve Loughran  
          17.
          		 AWS SDK V2 - Upgrade SDK to 2.20.28 and restores multipart copy Sub-task Resolved Ahmar Suhail  
          18.
          		 AWS SDK v2 typo in aws.evenstream.version Sub-task Resolved Ahmar Suhail  
          19.
          		 AWS SDK V2 - AuditFailureExceptions aren't being translated properly Sub-task Resolved Steve Loughran  

          Activity

            People

              ahmar Ahmar Suhail
              xiaoweisun xiaowei sun
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m