Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12723

S3A: Add ability to plug in any AWSCredentialsProvider

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.1
    • 2.8.0, 3.0.0-alpha1
    • fs/s3
    • None
    • Users can integrate a custom credential provider with S3A. See documentation of configuration property fs.s3a.aws.credentials.provider for further details.

    Description

      Although S3A currently has built-in support for org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider, com.amazonaws.auth.InstanceProfileCredentialsProvider, and org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider, it does not support any other credentials provider that implements the com.amazonaws.auth.AWSCredentialsProvider interface. Supporting the ability to plug in any com.amazonaws.auth.AWSCredentialsProvider instance will expand the options for S3 credentials, such as:

      • temporary credentials from STS, e.g. via com.amazonaws.auth.STSSessionCredentialsProvider
      • IAM role-based credentials, e.g. via com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider
      • a custom credentials provider that satisfies one's own needs, e.g. bucket-specific credentials, user-specific credentials, etc.

      To support this, we can add a configuration for the fully qualified class name of a credentials provider, to be loaded by S3AFileSystem.initialize(URI, Configuration).

      The configured credentials provider should implement com.amazonaws.auth.AWSCredentialsProvider and have a constructor that accepts (URI uri, Configuration conf).

      Attachments

        1. HADOOP-12723-branch-2-006.patch
          13 kB
          Steve Loughran
        2. HADOOP-12723.5.patch
          13 kB
          Steven K. Wong
        3. HADOOP-12723.4.patch
          10 kB
          Steven K. Wong
        4. HADOOP-12723.3.patch
          10 kB
          Steven K. Wong
        5. HADOOP-12723.2.patch
          9 kB
          Steven K. Wong
        6. HADOOP-12723.1.patch
          9 kB
          Steven K. Wong
        7. HADOOP-12723.0.patch
          9 kB
          Steven K. Wong

        Issue Links

          Activity

            People

              slider Steven K. Wong
              slider Steven K. Wong
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: