Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15763

Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.1
    • fs/azure
    • None

    Description

      ABFS phase II: address issues which surface in the field; tune things which need tuning, add more tests where appropriate. Improve docs, especially troubleshooting. Classpaths. The usual.

      Attachments

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-15407 Support Windows Azure Storage - Blob file system in Hadoop

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17736 Über-JIRA: abfs phase III: Hadoop 3.3.2 features & fixes

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16353 Backport ABFS changes from trunk to branch-3.2

          • Major - Major loss of function.
          • Resolved
          1.
          		 ABFS checkException to map 403 to AccessDeniedException Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          2.
          		 intermittent failure of TestAbfsClient.validateUserAgent Sub-task Resolved Bilahari T H  
          3.
          		 ABFS: Better exception handling of DNS errors Sub-task Resolved Da Zhou  
          4.
          		 ABFS: validate packing, transient classpath, hadoop fs CLI Sub-task Resolved Steve Loughran  
          5.
          		 Document ABFS Sub-task Resolved Thomas Marqardt  
          6.
          		 typo in AzureBlobFileSystem.getIsNamespaceEnabeld Sub-task Resolved Abhishek Modi  
          7.
          		 Make HTTPS the default protocol for ABFS Sub-task Resolved Da Zhou  
          8.
          		 ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console Sub-task Resolved Da Zhou  
          9.
          		 ABFS: Stop requiring client ID and tenant ID for MSI Sub-task Resolved Da Zhou  
          10.
          		 ABFS: Enable some tests for namespace not enabled account using OAuth Sub-task Resolved Da Zhou  
          11.
          		 ABFS: change createRemoteFileSystemDuringInitialization default to true Sub-task Resolved Sean Mackrory  
          12.
          		 ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. Sub-task Resolved junhua gu  
          13.
          		 ABFS: Throw IllegalArgumentException when Directory/File name ends with a period(.) Sub-task Resolved Shweta  
          14.
          		 Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore Sub-task Resolved Da Zhou  
          15.
          		 ABFS: make retry policy configurable Sub-task Resolved Da Zhou  
          16.
          		 ABFS: For HNS account, avoid unnecessary get call when doing Rename Sub-task Resolved Da Zhou  
          17.
          		 ABFS: replace $superuser with local user Sub-task Resolved Da Zhou  
          18.
          		 ABFS: Enable owner and group conversion for MSI and login user using OAuth Sub-task Resolved Da Zhou  
          19.
          		 ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 Sub-task Resolved junhua gu  
          20.
          		 ABFS: add try catch for UGI failure when initializing ABFS Sub-task Resolved Da Zhou  
          21.
          		 ABFS: getNamespaceEnabled can fail blocking user access thru ACLs Sub-task Resolved Da Zhou  
          22.
          		 ABFS: reduce list page size to to 500 Sub-task Resolved Da Zhou  
          23.
          		 ABFS: remove timeout check for DELETE and RENAME Sub-task Resolved Da Zhou  
          24.
          		 ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException Sub-task Resolved Da Zhou  
          25.
          		 ABFS: Bug fix for tolerateOobAppends configuration Sub-task Resolved Da Zhou  
          26.
          		 UserAgent string for ABFS Sub-task Resolved Shweta  
          27.
          		 ABFS: Better exception handling of DNS errors followup Sub-task Resolved Da Zhou  
          28.
          		 ABFS: Fix Date format parser Sub-task Resolved Da Zhou  
          29.
          		 ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store Sub-task Resolved Steve Loughran  
          30.
          		 NativeAzureFileSystem does not support setXAttr Sub-task Resolved Clemens Wolff  
          31.
          		 Failure of ABFS test ITestAbfsIdentityTransformer Sub-task Resolved Steve Loughran  
          32.
          		 Wasb tests to downgrade to skip when test a/c is namespace enabled Sub-task Resolved Masatake Iwasaki

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          33.
          		 ABFS: Should only transform username to short name Sub-task Resolved Da Zhou  
          34.
          		 hadoop fs mkdir / of nonexistent abfs container raises NPE Sub-task Resolved Gabor Bota  
          35.
          		 NPE in ABFS Client Credential Auth Sub-task Resolved Steve Loughran  
          36.
          		 NPE in setup/teardown of ITestAbfsDelegationTokens Sub-task Resolved Steve Loughran  
          37.
          		 ABFS: Bug fix for getPathProperties Sub-task Resolved Da Zhou  
          38.
          		 Update abfs storage back-end with "close" flag when application is done writing to a file Sub-task Resolved Vishwajeet Dusane  
          39.
          		 ABFS: add bufferpool to AbfsOutputStream Sub-task Resolved Da Zhou  
          40.
          		 ABFS: add FSMainOperationsBaseTest Sub-task Resolved Da Zhou  
          41.
          		 ABFS: add listFileStatus with StartFrom Sub-task Resolved Da Zhou  
          42.
          		 ABFS: transform full UPN for named user in AclStatus Sub-task Resolved Da Zhou  
          43.
          		 ABFS driver continues to retry on IOException responses from REST operations Sub-task Resolved Robert Levas  
          44.
          		 ABFS: Override access() to no-op for now Sub-task Resolved Da Zhou  
          45.
          		 ABFS: port Azure doc to 3.2 branch Sub-task Resolved Masatake Iwasaki  
          46.
          		 ABFS default blocksize change(256MB from 512MB) Sub-task Resolved Arun Singh  
          47.
          		 ABFS: Implement FileSystem.access() method Sub-task Resolved Bilahari T H  
          48.
          		 ABFS FileStatus.getModificationTime returns localized time instead of UTC Sub-task Resolved Bilahari T H  
          49.
          		 AzureADAuthenticator cannot authenticate in china Sub-task Resolved Unassigned  
          50.
          		 Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property Sub-task Resolved Gabor Bota  
          51.
          		 ABFS: Add missing query parameter for getPathStatus Sub-task Resolved Adam Antal  
          52.
          		 ABFS: Config to enable/disable flush operation Sub-task Resolved Sneha Vijayarajan  
          53.
          		 ABFS: fileSystemExists() should not call container level apis Sub-task Resolved Sneha Vijayarajan  
          54.
          		 Make AAD endpoint configurable on all Auth flows Sub-task Resolved Bilahari T H  
          55.
          		 Track Azure Blob File System client-perceived latency Sub-task Resolved Jeetesh Mangwani  
          56.
          		 Backport HADOOP-16548 - "ABFS: Config to enable/disable flush operation" to branch-2 Sub-task Resolved Sneha Vijayarajan  
          57.
          		 Backport HADOOP-16578 - "ABFS: fileSystemExists() should not call container level apis" to Branch-2 Sub-task Resolved Sneha Vijayarajan  
          58.
          		 Backport HADOOP-16587 - "Make AAD endpoint configurable on all Auth flows" to branch-2 Sub-task Resolved Bilahari T H  
          59.
          		 ABFS: add missing docs for configuration Sub-task Resolved Bilahari T H  
          60.
          		 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable Sub-task Resolved Sneha Vijayarajan  
          61.
          		 Remove unnecessary ABFS toString() invocations Sub-task Resolved Bilahari T H  
          62.
          		 ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account Sub-task Resolved Sneha Vijayarajan  
          63.
          		 ABFS: Enhance driver debug logs Sub-task Resolved Sneha Vijayarajan  
          64.
          		 ABFS: Add additional tests for CustomTokenProvider Sub-task Resolved Sneha Vijayarajan  
          65.
          		 ITestClientUrlScheme fails for accounts which don't support HTTP Sub-task Resolved Steve Loughran  
          66.
          		 NPE in UGI.getCurrentUser in ITestAbfsIdentityTransformer setup Sub-task Resolved Steve Loughran  
          67.
          		 Backport HADOOP-16612 - "Track Azure Blob File System client-perceived latency" to branch-2 Sub-task Resolved Bilahari T H  
          68.
          		 Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2 Sub-task Resolved Bilahari T H  
          69.
          		 Backport HADOOP-16548 - ABFS: Config to enable/disable flush operation issue to branch-3.2 Sub-task Resolved Sneha Vijayarajan  
          70.
          		 ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2 Sub-task Resolved Sneha Vijayarajan  
          71.
          		 Backport HADOOP-16612 "Track Azure Blob File System client-perceived latency" to branch-2 Sub-task Resolved Jeetesh Mangwani  
          72.
          		 Improve wasb and abfs resilience on double close() calls Sub-task Resolved Steve Loughran  
          73.
          		 ABFS: Combine append+flush calls for blockblob & appendblob Sub-task Resolved Ishani  
          74.
          		 ITestAzureBlobFileSystemCheckAccess failing Sub-task Resolved Bilahari T H  
          75.
          		 ABFS: update abfs.md to include config keys for identity transformation Sub-task Resolved Karthick Narendran  
          76.
          		 ITestAbfsClient.testContinuationTokenHavingEqualSign failing Sub-task Resolved Sneha Vijayarajan  
          77.
          		 ABFS: Send error back to client for Read Ahead request failure Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          78.
          		 ABFS: hadoop-dist fails to add wildfly in class path for hadoop-azure Sub-task Resolved Bilahari T H  
          79.
          		 ABFS: Optimize HttpRequest retry triggers Sub-task Resolved Sneha Vijayarajan  
          80.
          		 ABFS: Add unbuffer support to AbfsInputStream Sub-task Resolved Sahil Takiar  
          81.
          		 ABFS: Change in expiry calculation for MSI token provider Sub-task Resolved Bilahari T H  
          82.
          		 ABFS: Support for OAuth v2.0 endpoints Sub-task Resolved Bilahari T H  
          83.
          		 ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance Sub-task Resolved Bilahari T H  
          84.
          		 ABFS: Make list page size configurable Sub-task Resolved Bilahari T H  
          85.
          		 ABFS: Change in User-Agent header Sub-task Resolved Bilahari T H  
          86.
          		 Backport HADOOP-16890- "ABFS: Change in expiry calculation for MSI token provider" & HADOOP-16825 "ITestAzureBlobFileSystemCheckAccess failing" to branch-2 Sub-task Resolved Bilahari T H  
          87.
          		 ABFS: Support infinite lease dirs Sub-task Resolved Billie Rinaldi

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10h 50m
          88.
          		 ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) Sub-task Resolved Mehakmeet Singh  
          89.
          		 ABFS: Upgrade Store REST API Version to 2019-12-12 Sub-task Resolved Sneha Vijayarajan  
          90.
          		 ABFS: Avoid storage calls to check if the account is HNS enabled or not Sub-task Resolved Bilahari T H  
          91.
          		 ABFS: Improve the ABFS driver documentation Sub-task Resolved Bilahari T H  
          92.
          		 ABFS: Make PUT and POST operations idempotent Sub-task Resolved Sneha Vijayarajan  
          93.
          		 Support disabling buffered reads in ABFS positional reads Sub-task Resolved Anoop Sam John

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 9h 10m
          94.
          		 ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting Sub-task Resolved Sneha Vijayarajan  
          95.
          		 ABFS: Fix idempotency test failures when SharedKey is set as AuthType Sub-task Resolved Sneha Vijayarajan  
          96.
          		 Support for Appendblob in abfs driver Sub-task Resolved Ishani  
          97.
          		 javadoc failing in the yetus report with the latest trunk Sub-task Resolved Unassigned  
          98.
          		 ABFS: Fix the parsing errors in ABFS Driver with creation Time (being returned in ListPath) Sub-task Resolved Bilahari T H  
          99.
          		 ABFS: Long waits and unintended retries when multiple threads try to fetch token using ClientCreds Sub-task Resolved Bilahari T H  
          100.
          		 ABFS: GetAccessToken unrecoverable failures are being retried Sub-task Resolved Bilahari T H  
          101.
          		 ABFS: Fix For Idempotency code Sub-task Resolved Sneha Vijayarajan  
          102.
          		 ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic Sub-task Resolved Bilahari T H  
          103.
          		 ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run with SharedKey Sub-task Resolved Bilahari T H  
          104.
          		 ABFS: Test failure: Disable ITestAzureBlobFileSystemDelegationSAS tests Sub-task Resolved Bilahari T H  
          105.
          		 ABFS: Add debug log for rename failures Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          106.
          		 ABFS: configure output stream thread pool Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          107.
          		 ABFS: Enable checkaccess API Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          108.
          		 ABFS: Run the integration tests with various combinations of configurations and publish consolidated results Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 14h
          109.
          		 ABFS: Support for conditional overwrite Sub-task Resolved Sneha Vijayarajan  
          110.
          		 ABFS: Support for Client Correlation ID Sub-task Resolved Unassigned

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 40m
          111.
          		 ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          112.
          		 ABFS: Allow Random Reads to be of Buffer Size Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          113.
          		 ABFS: read-ahead error reporting breaks buffer management Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          114.
          		 ABFS: Logs should redact SAS signature Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m
          115.
          		 Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2 Sub-task Resolved Sally Zuo  
          116.
          		 Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" and HADOOP-16785. "Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix." to branch-2.10 Sub-task Resolved Sally Zuo  
          117.
          		 ABFS: read/cache footer with fs.azure.footer.read.request.size Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 12h 50m
          118.
          		 ABFS: testRenameFileOverExistingFile Fails after Contract test update Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          119.
          		 ABFS: SAS Test updates for version and permission update Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          120.
          		 ABFS: Piggyback flush on Append calls for short writes Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h
          121.
          		 ABFS: Delete Idempotency handling can lead to NPE Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          122.
          		 ABFS: Release Elastic ByteBuffer pool memory at outputStream close Sub-task Resolved Sneha Vijayarajan

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          123.
          		 ABFS: Set default ListMaxResults to max server limit Sub-task Resolved Thomas Marqardt

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 20m
          124.
          		 Optimise abfs incremental listings Sub-task Resolved Unassigned  
          125.
          		 ABFS: Fix auth failures getting counted as throttling errors in ABFS throttling interceptor Sub-task Resolved Unassigned  
          126.
          		 ABFS: Fix boundary conditions in InputStream seek and skip Sub-task Resolved Unassigned

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 50m
          127.
          		 ABFS: ITestAzureBlobFileSystemCheckAccess test failure if test doesn't have oauth keys Sub-task Resolved Steve Loughran

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          128.
          		 ABFS: Suport for customer provided encryption key Sub-task Resolved Bilahari T H

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 13h
          129.
          		 Correct abfs test assertion reversed in HADOOP-13327 Sub-task Resolved Sumangala Patki

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          130.
          		 ABFS: Toggle Store Mkdirs request overwrite parameter Sub-task Resolved Sumangala Patki

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          131.
          		 ABFS: Disable throttling update for auth failures Sub-task Resolved Sumangala Patki

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m

          Activity

            People

              DanielZhou Da Zhou
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 105h 20m
                  105h 20m