Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Hello,
When calling blobStore.blobMetadata(container, objectName).getSize() with a blobStoreContext using ApacheHCHttpCommandExecutorServiceModule, the returned BlobMetadata object has a size attribute which is null.
Issue spotted while using jclouds 2.1.2 over an S3 object store (Scality, but seems like it is reproducible with any other S3 provider, and potentially others OS providers like GCS, to be confirmed).
In essence, the minimal reproducer pseudo-code is:
BlobStoreContext context = ContextBuilder.newBuilder(provider) .credentials(identity, credential) .endpoint(endpoint) // usage of ApacheHCHttp module .modules(ImmutableList.<Module> of(new ApacheHCHttpCommandExecutorServiceModule(), new SLF4JLoggingModule())) .buildView(BlobStoreContext.class); try { BlobStore blobStore = context.getBlobStore(); // output is: "File size from metadata: null" System.out.println("File size from metadata: " + blobStore.blobMetadata(DUMMY_CONTAINER, DUMMY_FILE_NAME).getSize()); } finally { context.close(); }
For convenience, I've attached a ready to run reproducer to this ticket, here's how to run it:
> mkdir reproducer > tar xvzf jclouds_null_size_reproducer.tar.gz -C reproducer > cd reproducer > mvn install # note: pre-req from this stage is to have an S3 Object Store available, you can use minio one if you have docker (docker run -it -p 9000:9000 -v /mnt/data:/data minio/minio server /data) # replace with your Object Store endpoint, identity and access key > java -jar target/blobstore-basics-jar-with-dependencies.jar s3 http://10.66.33.1:9000 <identity here> <access key here> File size from metadata: 8 File size from metadata (with ApacheHCHttpCommandExecutorServiceModule): null
The reproducer code has jclouds-wire logs enabled (inside log folder) and I've also added the log files inside the attached tarball for convenience. We see that in the HTTP HEAD response (the 2nd one, which correspond to my .blobMetadata() call using ApacheHCHttpCommandExecutorServiceModule), the Content-* headers are missing :
>cat log/jclouds-wire.log # note: first .blobMetadata() call without ApacheHCHttpCommandExecutorServiceModule, the response contains Content-* headers 2019-07-31 17:30:13,257 DEBUG [jclouds.headers] [main] >> HEAD http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul 2019 15:30:13 GMT 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Authorization: AWS 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE= 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Server: MinIO/RELEASE.2019-07-24T02-02-23Z 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << ETag: "35a41457219c775659b6018fb7f465a1-1" 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Xss-Protection: 1; mode=block 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Security-Policy: block-all-mixed-content 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Vary: Origin 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, 31 Jul 2019 15:30:13 GMT 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul 2019 15:30:13 GMT 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: 15B687995977CC49 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Type: application/unknown 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Length: 8 # note: second call with ApacheHCHttpCommandExecutorServiceModule, the reponse is missing the content-* headers 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> HEAD http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul 2019 15:30:13 GMT 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Authorization: AWS 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE= 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Content-Security-Policy: block-all-mixed-content 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << ETag: "35a41457219c775659b6018fb7f465a1-1" 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, 31 Jul 2019 15:30:13 GMT 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Server: MinIO/RELEASE.2019-07-24T02-02-23Z 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Vary: Origin 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: 15B6879961644613 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Xss-Protection: 1; mode=block 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul 2019 15:30:13 GMT
I suspect the issue is coming from ApacheHCHttpCommandExecutorService.java#invoke method, it seems to be expecting the content-* headers are exclusively coming from the reponse.getEntity() (which I believe is null in case of a response to a HEAD request), because it strips it out of the headers list (which are well containing the content-* headers before the call to "filterOutContentHeaders"). A possible fix to cope with all requests kinds could be to strip it out only if it is already part of the payload (which I guess was the initial intent).
Environment details:
- jclouds version: 2.1.2 (but it is probably reproducible on older versions, to be confirmed)
- Operating system: Linux (kernel 4.15.0-55)
- Storage provider: S3 (reproduced on both Scality and MinIO), probably reproducible with other storage providers involving HTTP HEAD requests to get the metadata (GCS, ..)
- Java version:
>java -version
java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)