Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619

Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • fs/s3
    • None

    Description

      Features for S3Guard for Hadoop 3.3. Goal: take the experimental tag off

      Attachments

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13980 S3Guard CLI: Add fsck check and fix commands

          • Major - Major loss of function.
          • In Progress
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16646 Backport S3A enhancements and fixes from trunk to branch-3.2

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features

          • Major - Major loss of function.
          • Resolved
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15226 Über-JIRA: S3Guard Phase III: Hadoop 3.2 features

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1698

          1.
          		 add bulk delete call to metastore API & DDB impl Sub-task Resolved Unassigned  
          2.
          		 s3guard DynamoDB resource not found: tables not ACTIVE state after initial connection Sub-task Resolved Unassigned  
          3.
          		 S3Guard to support creating on-demand DDB tables Sub-task Resolved Steve Loughran  
          4.
          		 s3guard set-capacity to not retry on an access denied exception Sub-task Resolved Steve Loughran  
          5.
          		 S3Guard: Provide custom FileSystem Statistics. Sub-task Resolved Unassigned  
          6.
          		 s3guard set-capacity command to fail fast if bucket is unguarded Sub-task Resolved Gabor Bota  
          7.
          		 S3Guard, MetadataStore to support atomic create(path, overwrite=false) Sub-task Resolved Unassigned  
          8.
          		 S3AFileSystem.innerRename() to size metadatastore lists better Sub-task Resolved Unassigned  
          9.
          		 S3Guard: remove "experimental" from documentation Sub-task Resolved Gabor Bota  
          10.
          		 S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete() Sub-task Resolved Steve Loughran  
          11.
          		 S3Guard store becomes inconsistent after partial failure of rename Sub-task Resolved Steve Loughran  
          12.
          		 S3Guard: make short-circuit getFileStatus() configurable Sub-task Resolved Gabor Bota  
          13.
          		 hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard Sub-task Resolved Steve Loughran  
          14.
          		 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing Sub-task Resolved Gabor Bota  
          15.
          		 s3guard bucket-info will create s3guard table if FS is set to do this automatically Sub-task Resolved Gabor Bota  
          16.
          		 ITestS3GuardTool failures in testDestroyNoBucket() Sub-task Resolved Gabor Bota  
          17.
          		 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants Sub-task Resolved Gabor Bota  
          18.
          		 S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets Sub-task Resolved Gabor Bota  
          19.
          		 ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs caching is disabled Sub-task Resolved Gabor Bota  
          20.
          		 LocalMetadataStore put() does not retain isDeleted in parent listing Sub-task Resolved Gabor Bota  
          21.
          		 ITestS3AEmptyDirectory#testDirectoryBecomesEmpty fails when running with dynamo Sub-task Resolved Gabor Bota  
          22.
          		 ITestS3GuardListConsistency#testConsistentListAfterDelete fails when running with dynamo Sub-task Resolved Unassigned  
          23.
          		 NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard Sub-task Resolved Gabor Bota  
          24.
          		 Intermittent failures of some S3A tests with S3Guard in parallel test runs Sub-task Resolved Unassigned  
          25.
          		 DynamoDB table Update can fail S3A FS init Sub-task Resolved Steve Loughran  
          26.
          		 s3guard init and destroy command will create/destroy tables if ddb.table & region are set Sub-task Resolved Gabor Bota  
          27.
          		 S3Guard testConcurrentTableCreations to set r & w capacity == 0 Sub-task Resolved lqjacklee  
          28.
          		 s3guard bucket-info command to not print a stack trace on bucket-not-found Sub-task Resolved Adam Antal  
          29.
          		 ITestDynamoDBMetadataStore can leak (large) DDB tables in test failures/timeout Sub-task Resolved Steve Loughran  
          30.
          		 ITestDynamoDBMetadataStore should check if test ddb table set properly before initializing the test Sub-task Resolved Gabor Bota  
          31.
          		 Should be able to set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative directory listings Sub-task Resolved Gabor Bota  
          32.
          		 S3Guard: Better support for out-of-band operations Sub-task Resolved Gabor Bota  
          33.
          		 IndexOutOfBoundsException in ITestS3GuardToolLocal Sub-task Resolved Adam Antal  
          34.
          		 S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite Sub-task Resolved Ben Roling  
          35.
          		 S3Guard to support on-demand DDB tables Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          36.
          		 S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker Sub-task Resolved Gabor Bota  
          37.
          		 S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren Sub-task Resolved Gabor Bota  
          38.
          		 ITestS3GuardToolDynamoDB test failures Sub-task Resolved Steve Loughran  
          39.
          		 S3Guard: fail write that doesn't update metadata store Sub-task Resolved Ben Roling  
          40.
          		 Add a way to declare a toplevel path in a non-auth FS as auth Sub-task Resolved Sean Mackrory  
          41.
          		 Use configurable dynamo table name prefix in S3Guard tests Sub-task Resolved Ben Roling  
          42.
          		 transient failure of ITestS3GuardToolDynamoDB.testDestroyUnknownTable Sub-task Resolved Unassigned  
          43.
          		 S3Guard: Implement time-based (TTL) expiry for entries (and tombstones) Sub-task Resolved Gabor Bota  
          44.
          		 S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened Sub-task Resolved Steve Loughran  
          45.
          		 S3Guard: Remove LocalMetadataStore Sub-task Resolved Unassigned  
          46.
          		 "Wrong Billing mode" test failure in ITestS3GuardToolDynamoDB Sub-task Resolved Steve Loughran  
          47.
          		 DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry Sub-task Resolved Gabor Bota  
          48.
          		 S3Guard DDB store prune() doesn't translate AWS exceptions to IOEs Sub-task Resolved Steve Loughran  
          49.
          		 S3Guard table destroy to map IllegalArgumentExceptions to IOEs Sub-task Resolved Steve Loughran  
          50.
          		 S3A list operation doesn't pick up etags from results Sub-task Resolved Steve Loughran  
          51.
          		 ITestS3AMetadataPersistenceException failure Sub-task Resolved Unassigned  
          52.
          		 S3A tombstones can confuse empty directory status Sub-task Resolved Steve Loughran  
          53.
          		 Pass ITtlTimeProvider instance in initialize method in MetadataStore interface Sub-task Resolved Gabor Bota  
          54.
          		 S3A: Avoid inconsistencies between DDB and S3 Sub-task Resolved Steve Loughran  
          55.
          		 S3Guard Diff tool to list+ compare the etag and version fields Sub-task Resolved Unassigned  
          56.
          		 Hadoop S3Guard Prune command to support a -tombstone option. Sub-task Resolved Steve Loughran  
          57.
          		 ITestDynamoDBMetadataStore.testProvisionTable times out intermittently Sub-task Resolved Steve Loughran  
          58.
          		 mark DynamoDBMetadataStore.deleteTrackingValueMap as final Sub-task Resolved Kevin Su  
          59.
          		 Transient failure of ITestS3ARemoteFileChanged Sub-task Resolved Unassigned  
          60.
          		 ITestS3GuardOutOfBandOperations.deleteAfterTombstoneExpiryOobCreate failure Sub-task Resolved Unassigned  
          61.
          		 DynamoDBMetaStore deleteSubtree to delete leaf nodes first Sub-task Resolved Unassigned  
          62.
          		 S3AFilesystem.delete to incrementally update s3guard with deletions Sub-task Resolved Steve Loughran  
          63.
          		 S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren Sub-task Resolved Gabor Bota  
          64.
          		 S3Guard in auth mode doesn't raise AccessDeniedException on read of 0-byte file Sub-task Resolved Unassigned  
          65.
          		 findbugs warning on LocalMetadataStore.ttlTimeProvider sync Sub-task Resolved Steve Loughran  
          66.
          		 S3Guard ProgressiveRenameTracker to mark dest dir as authoritative on success Sub-task Resolved Steve Loughran  
          67.
          		 ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region Sub-task Resolved Steve Loughran  
          68.
          		 S3A to warn or fail if S3Guard is disabled Sub-task Resolved Gabor Bota  
          69.
          		 S3Guard operations log has tombstone/PUT swapped Sub-task Resolved Steve Loughran  
          70.
          		 Avoid/handle cached 404s during S3A file creation Sub-task Resolved Steve Loughran  
          71.
          		 s3guard auth path checks only check against unqualified source path Sub-task Resolved Steve Loughran  
          72.
          		 Race condition in DDB table init and waiting threads Sub-task Resolved Gabor Bota  
          73.
          		 s3guard prune command doesn't get AWS auth chain from FS Sub-task Resolved Steve Loughran  
          74.
          		 Region must be provided when requesting session credentials or SdkClientException will be thrown Sub-task Resolved Gabor Bota  
          75.
          		 ITestS3GuardDDBRootOperations. test_100_FilesystemPrune failure Sub-task Resolved Steve Loughran  
          76.
          		 ITestDynamoDBMetadataStoreScale fails when throttled. Sub-task Resolved Steve Loughran  
          77.
          		 s3guard bucket info to list a bit more about authoritative paths Sub-task Resolved Steve Loughran  
          78.
          		 audit/tune s3a authoritative flag in s3guard DDB Table Sub-task Resolved Steve Loughran  
          79.
          		 Prune -tombstones to remove children entries Sub-task Resolved Gabor Bota  

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h