Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
  3. HADOOP-19104

S3A HeaderProcessing to process all metadata entries of HEAD response

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • fs/s3
    • None

    Description

      S3A HeaderProcessing builds up an incomplete list of headers as its mapping of md to header. entries omits headers including
      x-amz-server-side-encryption-aws-kms-key-id

      proposed

      • review all headers which are stripped from "raw" responses and mapped into headers
      • make sure result of headers matches v1; looks like etags are different
      • make sure x-amz-server-side-encryption-aws-kms-key-id gets back
      • plus new checksum values

      v1 sdk

      
      # file: s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
      header.Content-Length="524671"
      header.Content-Type="binary/octet-stream"
      header.ETag="3e39531220fbd3747d32cf93a79a7a0c"
      header.Last-Modified="Tue Jan 02 00:15:13 GMT 2024"
      header.x-amz-server-side-encryption="AES256"
      
      

      v2 SDK. note how etag is now double quoted.

      
      # file: s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
      header.Content-Length="524671"
      header.Content-Type="binary/octet-stream"
      header.ETag=""3e39531220fbd3747d32cf93a79a7a0c""
      header.Last-Modified="Tue Jan 02 00:15:13 GMT 2024"
      header.x-amz-server-side-encryption="AES256"
      
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: