Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1516

First putblob should be signed with specific region rather than with default region during createcontainer API

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.2
    • Fix Version/s: None
    • Component/s: jclouds-blobstore
    • Labels:
      None
    • Environment:
      Linux
    • Flags:
      Patch

      Description

      When container in non default region exists, only first time put operation will be applied to find out whether bucket exists and have proper access or not. Aws sigv4 signature will be created based on the default region only as create bucket method was not honoring the region specified.

      So, Put request was first signed with default region and if the user does not have access to default(us-east-1) region,. Then it will be re-directed to sa-east-1 region, but the request is signed with us-east-1, the request is rejected and throws "The authorization header is malformed".

      Flow in case of user has access to default region:

      17:22:24.460 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
      17:22:24.460 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
      17:22:24.461 [bscThread-02] DEBUG jclouds.signature - << PUT
      /

      content-length:105
      content-type:text/xml
      host:test3.s3.amazonaws.com
      x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      x-amz-date:20190818T115218Z

      content-length;content-type;host;x-amz-content-sha256;x-amz-date
      f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      17:22:24.461 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256
      20190818T115218Z
      20190818/us-east-1/s3/aws4_request
      089a5248f5eff6e8b6378154acdf07bff7d208029c98c67af44c99b4a8f2df39
      17:22:24.463 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -1533211628: PUT https://test3.s3.amazonaws.com/ HTTP/1.1
      17:22:24.463 [bscThread-02] DEBUG jclouds.wire - >> "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>"
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> PUT https://test3.s3.amazonaws.com/ HTTP/1.1
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Host: test3.s3.amazonaws.com
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: 20190818T115218Z
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIGKQ7V52FQQJFYJQ/20190818/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=637d42fbf6684430ab0f08fd82cbae69f3261859e0031ad40054bccb829473da
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
      17:22:24.464 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
      17:22:25.671 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response -1533211628: HTTP/1.1 409 Conflict
      17:22:25.671 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 409 Conflict
      17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: chunked
      17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3
      17:22:25.672 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: 09E5163C51F25F34
      17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: WuN84GYMs47Nn6+48XYDpLZNvp0NPszokKyhxlzZk+ub8RhjbLpkfEI8E2tKWVCFKtJiXrhdpkc=
      17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 11:52:11 GMT
      17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << x-amz-bucket-region: sa-east-1
      17:22:25.673 [bscThread-02] DEBUG jclouds.headers - << Content-Type: application/xml
      17:22:25.673 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
      17:22:25.673 [bscThread-02] DEBUG jclouds.wire - << "<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>test3</BucketName><RequestId>09E5163C51F25F34</RequestId><HostId>WuN84GYMs47Nn6+48XYDpLZNvp0NPszokKyhxlzZk+ub8RhjbLpkfEI8E2tKWVCFKtJiXrhdpkc=</HostId></Error>"
      17:22:25.680 [bscThread-03] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking BucketExists
      17:22:25.681 [bscThread-03] DEBUG jclouds.signature - << HEAD
      /

      Flow in case of user has access to default region

      host:test3.s3-sa-east-1.amazonaws.com
      x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
      x-amz-date:20190818T115218Z

       

      Flow in case of user does not have access to default region.

      17:28:41.464 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
      17:28:41.464 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
      17:28:41.466 [bscThread-02] DEBUG jclouds.signature - << PUT
      /

      content-length:105
      content-type:text/xml
      host:test-poc-spectrum-scale.s3.amazonaws.com
      x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      x-amz-date:20190818T115835Z

      content-length;content-type;host;x-amz-content-sha256;x-amz-date
      f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      17:28:41.466 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256
      20190818T115835Z
      20190818/us-east-1/s3/aws4_request
      d8c68a44175c3b0b49182ca5658bff291b3f382d7d02b3fe25d6970912df1697
      17:28:41.467 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -541426552: PUT https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1
      17:28:41.467 [bscThread-02] DEBUG jclouds.wire - >> "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>"
      17:28:41.467 [bscThread-02] DEBUG jclouds.headers - >> PUT https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1
      17:28:41.467 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Host: test-poc-spectrum-scale.s3.amazonaws.com
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: 20190818T115835Z
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Authorization: AWS4-HMAC-SHA256 Credential=AKIA25YU33MFMAZO3B7F/20190818/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=8cf0d9f58bd0ba5fc607f5018e701cdf14587741169ba7deaaa7a9cad0548cb6
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
      17:28:41.468 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
      17:28:43.016 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response -541426552: HTTP/1.1 400 Bad Request
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 400 Bad Request
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: chunked
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Connection: close
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: 43F137234826AA08
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 11:58:29 GMT
      17:28:43.017 [bscThread-02] DEBUG jclouds.headers - << Content-Type: application/xml
      17:28:43.019 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
      17:28:43.019 [bscThread-02] DEBUG jclouds.wire - << "<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'sa-east-1'</Message><Region>sa-east-1</Region><RequestId>43F137234826AA08</RequestId><HostId>nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=</HostId></Error>"
      EXC:org.jclouds.aws.AWSResponseException: request PUT https://test-poc-spectrum-scale.s3.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='43F137234826AA08', requestToken='nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=', code='AuthorizationHeaderMalformed', message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'sa-east-1'', context='

      {Region=sa-east-1, HostId=nzJhRvv6j87VgSKV/mlGNhAqdietQCH4V5ArVUdIX2j1SnTChg2t1/GH20Zs3iz6Bc0MPzzAzwc=}

      '}

      I have a fix so that the first put operation honors the custom region specified and request will be signed with specific region rather than with default region. Tested with default region as well and worked well.

      After my fix, flow with stand-alone program is as below.

      17:36:09.043 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
      17:36:09.043 [bscThread-02] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking CreateBucket
      17:36:09.044 [bscThread-02] DEBUG jclouds.signature - << PUT
      /

      content-length:105
      content-type:text/xml
      host:test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com
      x-amz-content-sha256:f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      x-amz-date:20190818T120602Z

      content-length;content-type;host;x-amz-content-sha256;x-amz-date
      f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      17:36:09.044 [bscThread-02] DEBUG jclouds.signature - << AWS4-HMAC-SHA256
      20190818T120602Z
      20190818/sa-east-1/s3/aws4_request
      2996a3b4d85ab8c68e1378ab68db2f7d80b244969d1792391a49b9b7390bc920
      17:36:09.046 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -719121422: PUT https://test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com/ HTTP/1.1
      17:36:09.046 [bscThread-02] DEBUG jclouds.wire - >> "<CreateBucketConfiguration><LocationConstraint>sa-east-1</LocationConstraint></CreateBucketConfiguration>"
      17:36:09.046 [bscThread-02] DEBUG jclouds.headers - >> PUT https://test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com/ HTTP/1.1
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Host: test-poc-spectrum-scale.s3-sa-east-1.amazonaws.com
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> x-amz-content-sha256: f5d7dd57e1e23b516fc3543b9f24fc19a8409557905f48c6f412b3a67946ce96
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> X-Amz-Date: 20190818T120602Z
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Authorization: AWS4-HMAC-SHA256 Credential=AKIA25YU33MFMAZO3B7F/20190818/sa-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=982a3a62e85e03c68f4ac1e0da5cfa753cb81cf750a6fbff157b34681ed54774
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Type: text/xml
      17:36:09.047 [bscThread-02] DEBUG jclouds.headers - >> Content-Length: 105
      17:36:11.076 [bscThread-02] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response -719121422: HTTP/1.1 409 Conflict
      17:36:11.076 [bscThread-02] DEBUG jclouds.headers - << HTTP/1.1 409 Conflict
      17:36:11.076 [bscThread-02] DEBUG jclouds.headers - << Transfer-Encoding: chunked
      17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Server: AmazonS3
      17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-request-id: 5FB8D9C1B41E48EB
      17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-id-2: 54oqA5Lc+yl6Y9Ppz6kMd6hZs/iGFGfNFPqrOPX90Q731UH5KkYEYc7RS/4W9btahu0dxQKd3iI=
      17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Date: Sun, 18 Aug 2019 12:05:57 GMT
      17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << x-amz-bucket-region: sa-east-1
      17:36:11.077 [bscThread-02] DEBUG jclouds.headers - << Content-Type: application/xml
      17:36:11.077 [bscThread-02] DEBUG jclouds.wire - << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
      17:36:11.077 [bscThread-02] DEBUG jclouds.wire - << "<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>test-poc-spectrum-scale</BucketName><RequestId>5FB8D9C1B41E48EB</RequestId><HostId>54oqA5Lc+yl6Y9Ppz6kMd6hZs/iGFGfNFPqrOPX90Q731UH5KkYEYc7RS/4W9btahu0dxQKd3iI=</HostId></Error>"
      17:36:11.083 [bscThread-03] DEBUG o.j.rest.internal.InvokeHttpMethod -

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              didixith@in.ibm.com Dileep Dixith
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified