Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-7621

Fetcher has unclear error message on HEAD fetch stage or cache sizing

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.2.0
    • None
    • fetcher
    • None

    Description

      $ curl -L -v -O -s http://HOSTNAME_REDACTED/PATH_REDACTED.tar.gz
      *   Trying 172.17.4.10...
      * Connected to HOSTNAME_REDACTED (172.17.4.10) port 80 (#0)
      > GET /PATH_REDACTED.tar.gz HTTP/1.1
      > Host: HOSTNAME_REDACTED
      > User-Agent: curl/7.43.0
      > Accept: */*
      >
      < HTTP/1.1 302 FOUND
      < Server: nginx/1.4.6 (Ubuntu)
      < Date: Mon, 05 Jun 2017 17:58:04 GMT
      < Content-Type: text/html; charset=utf-8
      < Content-Length: 1947
      < Connection: keep-alive
      < Location: https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED?Signature=REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D
      <
      * Ignoring the response-body
      { [309 bytes data]
      * Connection #0 to host HOSTNAME_REDACTED left intact
      * Issue another request to this URL: 'https://BUCKET_REDACTED.s3.amazonaws.com:443/PATH_REDACTED.tar.gz?Signature=SIGNATURE_REDACTED%3D&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D'
      *   Trying 54.231.40.75...
      * Connected to BUCKET_REDACTED.s3.amazonaws.com (54.231.40.75) port 443 (#1)
      * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      * Server certificate: *.s3.amazonaws.com
      * Server certificate: DigiCert Baltimore CA-2 G2
      * Server certificate: Baltimore CyberTrust Root
      > GET /PATH_REDACTED.tar.gz?Signature=REDACTED&Expires=1496689084&AWSAccessKeyId=KEY_REDACTED&x-amz-security-token=TOKEN_REDACTED%3D HTTP/1.1
      > Host: BUCKET_REDACTED.s3.amazonaws.com
      > User-Agent: curl/7.43.0
      > Accept: */*
      >
      < HTTP/1.1 200 OK
      < x-amz-id-2: ID_REDACTED=
      < x-amz-request-id: REQUEST_ID_REDACTED
      < Date: Mon, 05 Jun 2017 17:58:07 GMT
      < Last-Modified: Thu, 01 Jun 2017 03:04:49 GMT
      < ETag: "ETAG_REDACTED"
      < Accept-Ranges: bytes
      < Content-Type: application/x-tar
      < Content-Length: 208245664
      < Server: AmazonS3
      <
      { [16360 bytes data]
      

      We have a micro-service which signs temporary urls for services which can't speak natively with S3. The above is an example download using curl. But when using the mesos fetcher the agent logs contain the following information:

      fetcher.cpp:479] Reverting to fetching directly into the sandbox for 'http://HOST_REDACTED/PATH_REDACTED.tar.gz', due to failure to fetch through the cache, with error: Could not determine size of cache file for 'USER_REDACTED@http://HOST_REDACTED/PATH_REDACTED.tar.gz' with error: No URL content-length available
      

      After some more investigation, it looks like the signing service had a bug where it would not sign the redirect URL for HEAD vs GET (only GET) so the HEAD request would fail with a 403. The source code erroneously reports this as missing Content-Length rather than a 403 error on the HEAD request.

      Attachments

        Activity

          People

            Unassigned Unassigned
            drcrallen Charles R Allen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: