Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-14556

S3A to support Delegation Tokens

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • fs/s3
    • None

    Description

      S3A to support delegation tokens where

      • an authenticated client can request a token via FileSystem.getDelegationToken()
      • Amazon's token service is used to request short-lived session secret & id; these will be saved in the token and marshalled with jobs
      • A new authentication provider will look for a token for the current user and authenticate the user if found

      This will not support renewals; the lifespan of a token will be limited to the initial duration. Also, as you can't request an STS token from a temporary session, IAM instances won't be able to issue tokens.

      Attachments

        1. HADOOP-14556.oath.patch
          14 kB
          Daryn Sharp
        2. HADOOP-14556.oath-002.patch
          15 kB
          Steve Loughran
        3. HADOOP-14556-001.patch
          87 kB
          Steve Loughran
        4. HADOOP-14556-002.patch
          87 kB
          Steve Loughran
        5. HADOOP-14556-003.patch
          116 kB
          Steve Loughran
        6. HADOOP-14556-004.patch
          104 kB
          Steve Loughran
        7. HADOOP-14556-005.patch
          112 kB
          Steve Loughran
        8. HADOOP-14556-007.patch
          225 kB
          Steve Loughran
        9. HADOOP-14556-008.patch
          310 kB
          Steve Loughran
        10. HADOOP-14556-009.patch
          327 kB
          Steve Loughran
        11. HADOOP-14556-010.patch
          360 kB
          Steve Loughran
        12. HADOOP-14556-010.patch
          360 kB
          Steve Loughran
        13. HADOOP-14556-011.patch
          364 kB
          Steve Loughran
        14. HADOOP-14556-012.patch
          371 kB
          Steve Loughran
        15. HADOOP-14556-013.patch
          408 kB
          Steve Loughran
        16. HADOOP-14556-014.patch
          412 kB
          Steve Loughran
        17. HADOOP-14556-015.patch
          427 kB
          Steve Loughran
        18. HADOOP-14556-016.patch
          444 kB
          Steve Loughran
        19. HADOOP-14556-017.patch
          476 kB
          Steve Loughran
        20. HADOOP-14556-018a.patch
          493 kB
          Steve Loughran
        21. HADOOP-14556-019.patch
          515 kB
          Steve Loughran
        22. HADOOP-14556-020.patch
          521 kB
          Steve Loughran
        23. HADOOP-14556-021.patch
          521 kB
          Steve Loughran
        24. HADOOP-14556-022.patch
          524 kB
          Steve Loughran
        25. HADOOP-14556-023.patch
          524 kB
          Steve Loughran
        26. HADOOP-14556-024.patch
          529 kB
          Steve Loughran
        27. HADOOP-14556-025.patch
          528 kB
          Steve Loughran
        28. HADOOP-14556-026.patch
          552 kB
          Steve Loughran
        29. HADOOP-14556-027.patch
          562 kB
          Steve Loughran
        30. HADOOP-14556-028.patch
          562 kB
          Steve Loughran
        31. HADOOP-14556-029.patch
          563 kB
          Steve Loughran

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-18330 S3AFileSystem removes Path when calling createS3Client

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          causes

          Sub-task - The sub-task of the issue HADOOP-16033 hamcrest-library declaration in hadoop-aws to be scoped test

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-16033 hamcrest-library declaration in hadoop-aws to be scoped test

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15806 Move ITestS3AMiniYarnCluster to S3A committers

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15091 S3aUtils.getEncryptionAlgorithm() always logs@Debug "Using SSE-C"

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          depends upon

          Sub-task - The sub-task of the issue HADOOP-15583 Stabilize S3A Assumed Role support

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14723 reinstate URI parameter in AWSCredentialProvider constructors

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15808 Harden Token service loader use

          • Major - Major loss of function.
          • Resolved
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13951 HDFS DelegationTokenFetcher can't print non-HDFS tokens in a tokenfile

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is blocked by

          Sub-task - The sub-task of the issue HADOOP-14833 Remove s3a user:secret authentication

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-16913 Support per-session S3 credentials

          • Major - Major loss of function.
          • Open
          is related to

          Sub-task - The sub-task of the issue HADOOP-15862 ABFS to support a Delegation Token provider which marshalls current Oauth secrets

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HADOOP-15921 UGI.createLoginUser to log token filename & token identifiers on load

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HADOOP-16164 S3aDelegationTokens to add accessor for tests to get at the token binding

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HADOOP-13276 S3a operations keep retrying if the password is wrong

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Sub-task - The sub-task of the issue HADOOP-15650 Add custom InstanceProfileCredentialsProvider with more resilience to throttling

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Sub-task - The sub-task of the issue HADOOP-16079 Token.toString faulting if any token listed can't load.

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16068 ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-12770 KMSClientProvider addDelegationTokens won't add if the credentials contain an expired one

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HBASE-20774 FSHDFSUtils#isSameHdfs doesn't handle S3 filesystems correctly.

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16658 S3A connector does not support including the token renewer in the token identifier

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14060 HDFS fetchdt command to return error codes on success/failure

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7154 TokenCache.obtainTokensForNamenodes() to get DTs even when security is off

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-15672 add s3guard CLI command to generate session keys for an assumed role

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-7149 javadocs for FileInputFormat and OutputFormat to mention DT collection

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          supercedes

          Sub-task - The sub-task of the issue HADOOP-15672 add s3guard CLI command to generate session keys for an assumed role

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: