[JCLOUDS-1549] Multipart upload fails on AWS S3 buckets with Object Lock and retention - ASF JIRA

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 2.2.0
Fix Version/s: None
Component/s: jclouds-blobstore
Labels:
None

Description

AWS S3 buckets with object lock and a default object retention period (governance/compliance) require the content MD5 header for PUT operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html)
When multipart uploading a blob into such a bucket, a response is returned indicating that the content MD5 header is missing.

The following snippet:

BlobStoreContext blobStoreContext = ContextBuilder.newBuilder("aws-s3")
    .endpoint("https://s3.amazonaws.com")
    .credentials("...", "...")
    .buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
ByteSource payloadByteSource = ByteSource.wrap("payloaddata".getBytes(StandardCharsets.UTF_8));
Blob blob = blobStore.blobBuilder("multipart-uploaded-blob")
    .userMetadata(Collections.emptyMap())
    .payload(payloadByteSource)
    .contentLength(payloadByteSource.size())
    .contentMD5(payloadByteSource.hash(Hashing.md5()))
    .build();
blobStore.putBlob("my-temp-object-locked-bucket", blob, PutOptions.Builder.multipart());
blobStoreContext.close();

Results in the exception:

com.google.common.util.concurrent.UncheckedExecutionException: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=', code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:395)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
	at io.model9.backup.common.objectstorage.MultipartMd5Test.multipartMd5(MultipartMd5Test.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=', code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy44.uploadPart(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:391)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at java.lang.Thread.getStackTrace(Thread.java:1559)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:393)
	... 25 more

Here are the requests (URL and headers) initiated by JClouds in the above snippet:

GET >> https://s3.amazonaws.com/my-temp-object-locked-bucket?location
{Host=[s3.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=db96f64af0bc4457ceda679805b6bb70656494b0ae5ae4d6501f1b19847dc31a]}
POST >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploads
{Content-Type=[application/unknown], Content-MD5=[nirLkofY1bN5hCACAirgRA==], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=fdb6a51c9cfbe116f8da033e079d4a9033a63b1d7a3abe8256546b2363f2dddb]}
PUT >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Content-Type=[application/unknown], Content-Length=[11], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[9e8f1b13c357dd00b99dd1760be67a3fdeb3ed1e3605307e549ab74c483a43fa], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=.., SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=9aec5a51bbffaa95321a054d8d6aad40b7c7d87d6967562592f8aaf51bb5e7aa]}
DELETE >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f2e53347d054d7c4a08eb1d62b1ff828d90135d0563d85ee8547626788dfd290]}

It looks like the blob's MD5 hash is added to the POST to initiate the multipart upload but is missing from the part's PUT itself.

Multipart upload fails on AWS S3 buckets with Object Lock and retention

Details

Description

Attachments

Activity

People

Dates