Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18067

Über-jira: S3A Hadoop 3.3.5 features

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.2
    • 3.3.5
    • fs/s3
    • None

    Description

      Everything for s3a connector in hadoop 3.3.5.

      followed up by HADOOP-18477

      Attachments

        Issue Links

          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features

          • Major - Major loss of function.
          • Open
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17566 Über-jira: S3A Hadoop 3.3.2 features

          • Major - Major loss of function.
          • Resolved
          1.
          		 ITestS3AContractSeek.teardown closes FS before superclass does its cleanup Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          2.
          		 fs.s3a.buffer.dir to be under Yarn container path on yarn applications Sub-task Resolved Monthon Klongklaew

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          3.
          		 remove misleading fs.s3a.delegation.tokens.enabled prompt Sub-task Resolved Unassigned

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          4.
          		 Improve isolation of FS instances in S3A committer tests Sub-task Resolved Unassigned  
          5.
          		 Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer Sub-task Resolved Steve Loughran  
          6.
          		 S3A to use a thread pool for async path operations Sub-task Resolved Unassigned  
          7.
          		 Distcp to set S3 Storage Class Sub-task Resolved Unassigned

          0%
          Original Estimate - 168h
          Remaining Estimate - 168h
          8.
          		 Add S3A support for Async Scatter/Gather IO Sub-task Resolved Mukund Thakur  
          9.
          		 S3 SSEC tests to downgrade when running against a mandatory encryption object store Sub-task Resolved Monthon Klongklaew

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          10.
          		 S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use Sub-task Resolved Ahmar Suhail

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h
          11.
          		 Support configuration of different S3 storage classes Sub-task Resolved Monthon Klongklaew

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          12.
          		 S3A to support Requester Pays Buckets Sub-task Resolved Daniel Carl Jones

          100%
          Original Estimate - 2h Original Estimate - 2h
          Time Spent - 6h 50m
          13.
          		 Upgrade AWS SDK to 1.12.132 Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 20m
          14.
          		 S3Object must be closed/ref set to null in S3AInputStream.closeStream() Sub-task Resolved Mukund Thakur  
          15.
          		 S3A to downgraded to unguarded if the DDB table cannot be found Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          16.
          		 Remove S3Guard - no longer needed Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 15h 20m
          17.
          		 S3A auditing leaks memory through ThreadLocal references Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 7h 40m
          18.
          		 Implement paging during S3 multi object delete. Sub-task Resolved Mukund Thakur

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 10m
          19.
          		 ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged Sub-task Resolved Daniel Carl Jones

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 10m
          20.
          		 distcp to use openFile() with sequential IO; ranges of reads Sub-task Resolved Steve Loughran  
          21.
          		 S3A storage class option only picked up when buffering writes to disk Sub-task Resolved Monthon Klongklaew

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          22.
          		 AWS SDK update to 1.12.262 to address jackson CVE-2018-7489 and AWS CVE-2022-31159 Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4.5h
          23.
          		 HEAD OBJECT returns only 400 BAD REQUEST when token is expired Sub-task Resolved Unassigned  
          24.
          		 ITestS3Select.testSelectSeekFullLandsat is timing out Sub-task Resolved Ahmar Suhail

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          25.
          		 ITestCustomSigner fails when access point name has '-' Sub-task Resolved Ahmar Suhail

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          26.
          		 s3a FS init logs at warn if fs.s3a.create.storage.class is unset Sub-task Resolved Viraj Jasani  
          27.
          		 ILoadTestS3ABulkDeleteThrottling failing Sub-task Resolved Ahmar Suhail

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          28.
          		 Fix FileSystem leak in ITestS3AAWSCredentialsProvider Sub-task Resolved Viraj Jasani  
          29.
          		 S3AInputStream.unbuffer() async drain not releasing http connections Sub-task Resolved Steve Loughran  
          30.
          		 ITestS3AIOStatisticsContext failure Sub-task Resolved Mehakmeet Singh  
          31.
          		 s3a endpoint per bucket configuration in pyspark is ignored Sub-task Resolved Unassigned  
          32.
          		 S3A FS to add "fs.s3a.create.performance" to the builder file creation option set Sub-task Resolved Steve Loughran  
          33.
          		 upgrade AWS SDK to 1.12.316 Sub-task Resolved Steve Loughran  
          34.
          		 Improve S3A committers documentation clarity Sub-task Resolved Daniel Carl Jones

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          35.
          		 Add some S3A-specific create file options Sub-task Resolved Steve Loughran  
          36.
          		 AvroFSInput opens a stream twice and discards the second one without closing Sub-task Resolved Steve Loughran  
          37.
          		 s3a magic committer may commit more data Sub-task Resolved Steve Loughran  
          38.
          		 S3A to support user-specified content encoding Sub-task Resolved Holden Karau

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6h 20m
          39.
          		 S3A server-side encryption tests fail before checking encryption tests should skip Sub-task Resolved Daniel Carl Jones  
          40.
          		 Leak of S3AInstrumentation instances via hadoop Metrics references Sub-task Resolved Steve Loughran  

          Activity

            People

              mthakur Mukund Thakur
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 170h Original Estimate - 170h
                  170h
                  Remaining:
                  Time Spent - 70h 40m Remaining Estimate - 168h
                  168h
                  Logged:
                  Time Spent - 70h 40m Remaining Estimate - 168h
                  70h 40m