|
|
|
HADOOP-17401
|
HADOOP-11694
GCS to support per-bucket configuration
|
Steve Loughran
|
t oo
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-14081
|
HADOOP-11694
S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
|
Rajesh Balamohan
|
Rajesh Balamohan
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-14019
|
HADOOP-11694
fix some typos in the s3a docs
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13871
|
HADOOP-11694
ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13857
|
HADOOP-11694
S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13855
|
HADOOP-11694
Fix a couple of the s3a statistic names to be consistent with the rest
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13823
|
HADOOP-11694
s3a rename: fail if dest file exists
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13801
|
HADOOP-11694
regression: ITestS3AMiniYarnCluster failing
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13772
|
HADOOP-11694
ITestS3AContractRootDir still playing up, bug in eventually() retry logic?
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13755
|
HADOOP-11694
Purge superfluous/obsolete S3A Tests
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13735
|
HADOOP-11694
ITestS3AFileContextStatistics.testStatistics() failing
|
Pieter Reuse
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13729
|
HADOOP-11694
switch to Configuration.getLongBytes for byte options
|
Abhishek Modi
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13727
|
HADOOP-11694
S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.
|
Chris Nauroth
|
Rajesh Balamohan
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13703
|
HADOOP-11694
S3ABlockOutputStream to pass Yetus & Jenkins
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13692
|
HADOOP-11694
hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.
|
Chris Nauroth
|
Chris Nauroth
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13680
|
HADOOP-11694
fs.s3a.readahead.range to use getLongBytes
|
Abhishek Modi
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13674
|
HADOOP-11694
S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.
|
Chris Nauroth
|
Chris Nauroth
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13655
|
HADOOP-11694
document object store use with fs shell and distcp
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13621
|
HADOOP-11694
s3:// should have been fully cut off from trunk
|
Mingliang Liu
|
Mingliang Liu
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13614
|
HADOOP-11694
Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13599
|
HADOOP-11694
s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13569
|
HADOOP-11694
S3AFastOutputStream to take ProgressListener in file create()
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13567
|
HADOOP-11694
S3AFileSystem to override getStorageStatistics() and so serve up its statistics
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13566
|
HADOOP-11694
NPE in S3AFastOutputStream.write
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13560
|
HADOOP-11694
S3ABlockOutputStream to support huge (many GB) file writes
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13541
|
HADOOP-11694
explicitly declare the Joda time version S3A depends on
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13540
|
HADOOP-11694
improve section on troubleshooting s3a auth problems
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13531
|
HADOOP-11694
S3A output streams to share a single LocalDirAllocator for round-robin drive use
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13405
|
HADOOP-11694
doc for “fs.s3a.acl.default” indicates incorrect values
|
Shen Yinjie
|
Shen Yinjie
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13336
|
HADOOP-11694
S3A to support per-bucket configuration
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13324
|
HADOOP-11694
s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints)
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13319
|
HADOOP-11694
S3A to list InstanceProfileCredentialsProvider after EnvironmentVariableCredentialsProvider
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13313
|
HADOOP-11694
S3A TemporaryAWSCredentialsProvider to support Hadoop Credential providers for secrets
|
Unassigned
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13310
|
HADOOP-11694
S3A reporting of file group as empty is harmful to compatibility for the shell.
|
Unassigned
|
Chris Nauroth
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13309
|
HADOOP-11694
Document S3A known limitations in file ownership and permission model.
|
Chris Nauroth
|
Chris Nauroth
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13286
|
HADOOP-11694
add a S3A scale test to do gunzip and linecount
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13281
|
HADOOP-11694
S3A to track multipart upload count, size, duration
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13275
|
HADOOP-11694
hadoop fs command path doesn't include translation of amazon client exceptions
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Invalid
|
|
|
|
|
|
|
|
HADOOP-13262
|
HADOOP-11694
set multipart delete timeout to 5 * 60s in S3ATestUtils.createTestFileSystem
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13259
|
HADOOP-11694
mvn fs/s3 test runs to set DNS TTL to 20s
|
Unassigned
|
Steve Loughran
|
|
Resolved |
Won't Fix
|
|
|
|
|
|
|
|
HADOOP-13252
|
HADOOP-11694
Tune S3A provider plugin mechanism
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13241
|
HADOOP-11694
document s3a better
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13237
|
HADOOP-11694
s3a initialization against public bucket fails if caller lacks any credentials
|
Chris Nauroth
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13231
|
HADOOP-11694
Isolate test path used by a few S3A tests for more reliable parallel execution.
|
Steve Loughran
|
Chris Nauroth
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13212
|
HADOOP-11694
Provide an option to set the socket buffers in S3AFileSystem
|
Rajesh Balamohan
|
Rajesh Balamohan
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13208
|
HADOOP-11694
S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13203
|
HADOOP-11694
S3A: Support fadvise "random" mode for high performance readPositioned() reads
|
Rajesh Balamohan
|
Rajesh Balamohan
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13188
|
HADOOP-11694
S3A file-create should throw error rather than overwrite directories
|
Steve Loughran
|
Raymie Stata
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13171
|
HADOOP-11694
Add StorageStatistics to S3A; instrument some more operations
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13162
|
HADOOP-11694
Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs
|
Rajesh Balamohan
|
Rajesh Balamohan
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13158
|
HADOOP-11694
S3AFileSystem#toString might throw NullPointerException due to null cannedACL.
|
Chris Nauroth
|
Chris Nauroth
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13154
|
HADOOP-11694
S3AFileSystem printAmazonServiceException/printAmazonClientException appear copy & paste of AWS examples
|
Steve Loughran
|
Steve Loughran
|
|
Closed |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13131
|
HADOOP-11694
Add tests to verify that S3A supports SSE-S3 encryption
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13130
|
HADOOP-11694
s3a failures can surface as RTEs, not IOEs
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13122
|
HADOOP-11694
Customize User-Agent header sent in HTTP requests by S3A.
|
Chris Nauroth
|
Chris Nauroth
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13113
|
HADOOP-11694
Enable parallel test execution for hadoop-aws.
|
Chris Nauroth
|
Chris Nauroth
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13106
|
HADOOP-11694
if fs.s3a.block.size option == 0, use partition size option for blocksize
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Won't Fix
|
|
|
|
|
|
|
|
HADOOP-13078
|
HADOOP-11694
switch hadoop-aws back to using the (heavy) amazon-sdk JAR
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Won't Fix
|
|
|
|
|
|
|
|
HADOOP-13062
|
HADOOP-11694
S3A Introspect to invoke incompatible AWS TransferManagerConfiguration methods
|
Unassigned
|
Steve Loughran
|
|
Resolved |
Won't Fix
|
|
|
|
|
|
|
|
HADOOP-13058
|
HADOOP-11694
S3A FS fails during init against a read-only FS if multipart purge is enabled
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13050
|
HADOOP-11694
Upgrade to AWS SDK 1.11.45
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13047
|
HADOOP-11694
S3a Forward seek in stream length to be configurable
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-13028
|
HADOOP-11694
add low level counter metrics for S3A; use in read performance tests
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-13009
|
HADOOP-11694
add option for lazy open() on s3a
|
Unassigned
|
Steve Loughran
|
|
Resolved |
Invalid
|
|
|
|
|
|
|
|
HADOOP-12997
|
HADOOP-11694
s3a to pass PositionedReadable contract tests, improve readFully perf.
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-12978
|
HADOOP-11694
move s3a to slf4j logging
|
Unassigned
|
Steve Loughran
|
|
Resolved |
Not A Problem
|
|
|
|
|
|
|
|
HADOOP-12977
|
HADOOP-11694
s3a to handle delete("/", true) robustly
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12976
|
HADOOP-11694
s3a toString to be meaningful in logs
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-12844
|
HADOOP-11694
Recover when S3A fails on IOException in read()
|
Pieter Reuse
|
Pieter Reuse
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12804
|
HADOOP-11694
Read Proxy Password from Credential Providers in S3 FileSystem
|
Larry McCay
|
Larry McCay
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12774
|
HADOOP-11694
s3a should use UGI.getCurrentUser.getShortname() for username
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12753
|
HADOOP-11694
S3A JUnit tests failing if using HTTP proxy
|
Zoran Rajic
|
Zoran Rajic
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12642
|
HADOOP-11694
Update documentation to cover fs.s3.buffer.dir enhancements
|
Unassigned
|
Jason Archip
|
|
Resolved |
Implemented
|
|
|
|
|
|
|
|
HADOOP-12444
|
HADOOP-11694
Support lazy seek in S3AInputStream
|
Rajesh Balamohan
|
Rajesh Balamohan
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12387
|
HADOOP-11694
Improve S3AFastOutputStream memory management
|
Steve Loughran
|
Thomas Demoor
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-12292
|
HADOOP-11694
Make use of DeleteObjects optional
|
Thomas Demoor
|
Thomas Demoor
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-12169
|
HADOOP-11694
ListStatus on empty dir in S3A lists itself instead of returning an empty list
|
Pieter Reuse
|
Pieter Reuse
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-11918
|
HADOOP-11694
Listing an empty s3a root directory throws FileNotFound.
|
Lei (Eddy) Xu
|
Lei (Eddy) Xu
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-11874
|
HADOOP-11694
s3a can throw spurious IOEs on close()
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Duplicate
|
|
|
|
|
|
|
|
HADOOP-11684
|
HADOOP-11694
S3a to use thread pool that blocks clients
|
Thomas Demoor
|
Thomas Demoor
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-11573
|
HADOOP-11694
verify that s3a handles / in secret key
|
Steve Loughran
|
Steve Loughran
|
|
Resolved |
Done
|
|
|
|
|
|
|
|
HADOOP-11262
|
HADOOP-11694
Enable YARN to use S3A
|
Pieter Reuse
|
Thomas Demoor
|
|
Resolved |
Fixed
|
|
|
|
|
|
|
|
HADOOP-10643
|
HADOOP-11694
Add NativeS3Fs that delegates calls from FileContext apis to native s3 fs implementation
|
Sumit Kumar
|
Sumit Kumar
|
|
Resolved |
Duplicate
|
|
|
|
|