Bulk Operation

HADOOP-18477 Assist reflection IO with WrappedOperations class

HADOOP-18477 S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd filenames

HADOOP-18477 fs.getXattrs(path) for S3FS doesn't have x-amz-server-side-encryption-aws-kms-key-id header.

HADOOP-18477 S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support

HADOOP-18477 ITestS3GuardTool fails if SSE/DSSE encryption is used

HADOOP-18477 remove/deprecate fs.s3a.multipart.purge

HADOOP-18477 Add ability to configure ConnectionTTL of http connections while creating S3 Client.

HADOOP-18477 Add a way to get the IOStatistics of active filesystems in long-lived processes

HADOOP-18477 Support Overwrite Directory On Commit For S3A Committers

HADOOP-18477 s3a client SSLException is raised after very long timeout "Unsupported or unrecognized SSL message"

HADOOP-18477 Upgrade aws-java-sdk to 1.12.499+

HADOOP-18477 ITestS3ACopyFromLocalFile: AuditFailureException

HADOOP-18477 Make it possible to disable ITestS3ATemporaryCredentials

HADOOP-18477 ITestS3AFileSystemStatistic failure on mvn verify

HADOOP-18477 Test failures with CSE enabled

HADOOP-18477 Upgrade aws-java-sdk to 1.12.367+

HADOOP-18477 Change fs.s3a.directory.marker.retention to "keep"

HADOOP-18477 ITestS3ABlockOutputArray failure with IO File name too long

HADOOP-18477 S3ABlockOutputStream doesn't collect stats on multipart block uploads

HADOOP-18477 S3A ITestS3ABucketExistence access point test failure

HADOOP-18477 S3A audit header to include count of items in delete ops

HADOOP-18477 Amazon S3 disabling ACLs on all new buckets

HADOOP-18477 Add new store vendor config option

HADOOP-18477 Add API for bulk/paged object deletion

HADOOP-18477 Add "versions" tool to s3a command line entry point

HADOOP-18477 improve s3a committer stats collected

HADOOP-18477 define s3a encryption behaviour on copy

HADOOP-18477 S3A AssumedRole credentials provider should use Instance Role credentials in chain for assuming role

HADOOP-18477 Warn when no region is configured

HADOOP-18477 New configuration for static headers to be added to all S3 requests

HADOOP-18477 S3A: add option to disable probe for dir marker recreation on delete/rename.

HADOOP-18477 support multiple s3a integration test runs on same bucket in parallel

HADOOP-18477 hadoop-aws tests to take a configurable subdir in the test bucket

HADOOP-18477 ITestS3APrefetchingInputStream does not skip if no CSV test file available

HADOOP-18477 ITestS3AInputStreamPerformance#testDecompressionSequential128K NPE if no CSV file available

HADOOP-18477 Remove fs.s3a.executor.capacity

HADOOP-18477 Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects

HADOOP-18477 Support AWS IAM Identity Centre (prev. AWS SSO) for providing credentials to S3A

HADOOP-18477 Improve S3A delegations token documentation

HADOOP-18477 Understand status of S3 access point alias support in S3A

HADOOP-18477 Public dataset class for S3A integration tests

HADOOP-18477 Address JavaDoc warnings in classes like MarkerTool, S3ObjectAttributes, etc.

HADOOP-18477 S3A Authentication to support WebIdentity

HADOOP-18477 remove filtering of directory markers in s3a RenameOperation

HADOOP-18477 S3A connector to improve support for all AWS partitions

HADOOP-18477 s3 and abfs incremental listing: use SAX parsers to stream results to list iterators

HADOOP-18477 Add s3a tool to convert S3 server logs to avro/csv files

HADOOP-18477 abfs & s3a FS instantiate triggers warning about deprecated io.bytes.per.checksum

HADOOP-18477 S3A DeleteOperation to parallelize POSTing of bulk deletes

HADOOP-18477 S3A: Allow SSE configurations per object path

HADOOP-18477 ITestCustomSigner fails with NPE against private endpoint

HADOOP-18477 S3A Xattr/getXAttr to handle directories without markers

HADOOP-18477 TestS3AGetFileStatus:testNotFound() to use intercept()

HADOOP-18477 s3a listing IOStatistics to count #of entries returned per LIST call

HADOOP-18477 compatibility table in directory_markers.md doesn't render right

HADOOP-18477 S3A doesn't calculate Content-MD5 on uploads

HADOOP-18477 Use S3 content-range header to update length of an object during reads

HADOOP-18477 GCS to support per-bucket configuration

HADOOP-18477 hadoop-cloud-storage transient dependencies need review

HADOOP-18477 S3A (async) ObjectListingIterator to block in hasNext() for results

HADOOP-18477 ITestS3AConfiguration.testProxyConnection failing when s3a bucket probe disabled

HADOOP-18477 S3A delegation token binding to support secondary binding list

HADOOP-18477 S3A deleteObjects hanging/retrying forever

HADOOP-18477 S3A client retries on SSL Auth exceptions triggered by "." bucket names

HADOOP-18477 log accepted/rejected fs.s3a.authoritative.path paths @ debug

HADOOP-18477 ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative

HADOOP-18477 NPE in s3a byte buffer block upload

HADOOP-18477 S3A DT marshalling to include nested error text in wrapped message

HADOOP-18477 Possible inconsistent state of AbstractDelegationTokenSecretManager

HADOOP-18477 AWS AssumedRoleCredentialProvider needs ExternalId add

HADOOP-18477 ITestDynamoDBMetadataStore.testTableVersioning failure -DDB deleteItem consistency?

HADOOP-18477 Add AWS S3 Transfer acceleration support

HADOOP-18477 Document `dynamodb:TagResource` an explicit client-side permission for S3Guard

HADOOP-18477 typo in TestNeworkBinding

HADOOP-18477 Tune hadoop-aws parallel test surefire/failsafe settings

HADOOP-18477 increase the default number of threads and http connections in S3A

HADOOP-18477 ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled

HADOOP-18477 S3A DT support to warn when loading expired token

HADOOP-18477 s3a to improve diags on s3a bad request message

HADOOP-18477 S3A Secret access to fall back to XML if credential provider raises IOE.

HADOOP-18477 S3AFilesystem trash handling should respect the current UGI

HADOOP-18477 Encrypt S3A buffered data on disk

HADOOP-18477 S3 Select Exceptions are not being converted to IOEs

HADOOP-18477 Review S3A documentation to make sure it is consistent with the current codebase

HADOOP-18477 Speed up S3A test runs

HADOOP-18477 ITestS3AMiniYarnCluster fails on sequential runs with Kerberos error

HADOOP-18477 ITestS3ARemoteFileChanged doesn't overwrite test data creation

HADOOP-18477 S3A openFile() options to allow etag/version to be set

HADOOP-18477 multipart/huge file upload tests to look at checksums returned

HADOOP-18477 ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup

HADOOP-18477 S3aDelegationTokens to add accessor for tests to get at the token binding

HADOOP-18477 S3AInputStream read(bytes[]) to not retry on read failure: pass action up

HADOOP-18477 ITestS3A select tests fail if user kinited in

HADOOP-18477 Add S3AWriteOpContext for write ops; pass in statistics and other settings

HADOOP-18477 Support multipart download in S3AFileSystem

HADOOP-18477 Some S3A tests leak filesystem instances

HADOOP-18477 Use lighter-weight alternatives to innerGetFileStatus where possible

HADOOP-18477 S3a DelegationToken bindings to to support a "correlation ID" for the UA header

HADOOP-18477 S3AInputStream logging to make it easier to debug file leakage

HADOOP-18477 Report problems w/ local S3A buffer directory meaningfully

HADOOP-18477 S3AInputStream.seek should throw EOFException if seeking past the end of file

HADOOP-18477 S3AInputStream.remainingInFile should use nextReadPos

HADOOP-18477 ITestS3AContractRootDir failure on non-S3Guarded bucket

HADOOP-18477 Clarify committers.md around v2 failure handling

HADOOP-18477 NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead

HADOOP-18477 Add custom InstanceProfileCredentialsProvider with more resilience to throttling

HADOOP-18477 Remove transient dependency on hadoop-hdfs-client

HADOOP-18477 S3A Filesystem does not check return from AmazonS3Client deleteObjects

HADOOP-18477 S3A to support configuring various AWS S3 client extended options

HADOOP-18477 Add a way for an FS instance to say "really, no trash interval at all"

HADOOP-18477 s3guard uploads command to list date and initiator of outstanding uploads

HADOOP-18477 S3AFileStatus to add a serialVersionUID; review & test serialization

HADOOP-18477 strip s3.amazonaws.com off hostnames before making s3a calls

HADOOP-18477 s3guard bucket-info command to add a verify-property <key>=<value> <bucket>

HADOOP-18477 S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy

HADOOP-18477 make s3a read fault injection configurable including "off"

HADOOP-18477 review S3A translateException translation matches IBM CORS spec

HADOOP-18477 S3AInputStream.skip() to use lazy seek

HADOOP-18477 test and document use of fs.s3a.signing-algorithm

HADOOP-18477 builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload

HADOOP-18477 support git-secrets commit hook to keep AWS secrets out of git

HADOOP-18477 s3guard bucket-info command to include default bucket encryption info

HADOOP-18477 s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet

HADOOP-18477 Add common getFileBlockLocations() emulation for object stores, including S3A

HADOOP-18477 Handle S3A "glacier" data

HADOOP-18477 clean up ITestS3AFileSystemContract

HADOOP-18477 AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade

HADOOP-18477 test YARN log collection works to s3a

HADOOP-18477 S3A input stream to support ByteBufferReadable

HADOOP-18477 Test MR split optimisation with recursive listing

HADOOP-18477 Filesystem discovery to stop loading implementation classes

HADOOP-18477 S3A to implement rename(final Path src, final Path dst, final Rename... options)

HADOOP-18477 S3a auth exception to link to a wiki page on the problem

HADOOP-18477 S3A: Set thread names with more specific information about the call.

HADOOP-18477 S3A can support short user-friendly aliases for configuration of credential providers.

HADOOP-18477 s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum

HADOOP-18477 FileSystem/s3a processDeleteOnExit to skip the exists() check

HADOOP-18477 shell rm command to not rename to ~/.Trash in object stores

HADOOP-18477 Test hadoop fs shell against s3a; fix problems

HADOOP-18477 add a special 0 byte input stream for empty blobs

HADOOP-18477 S3a operations keep retrying if the password is wrong

HADOOP-18477 cherry pick s3 ehancements from PrestoS3FileSystem

HADOOP-18477 Impersonate hosts in s3a for better data locality handling