Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12867

Content-Encoding for PutS3Object and FetchS3Object

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.25.0
    • None
    • Core Framework
    • None

    Description

      Hi,

      HTTP has content-encoding with gzip as a standard. See here: https://www.iana.org/assignments/http-parameters/http-parameters.xhtml

      By using AWS S3 or other S3 compliant storages users compress the content to be uploaded with gzip and set the HTTP Header Content-Encodig: gzip and the mime.type accordingly. 

      The PutS3Object processor does not have an attribute to set the Content-Encoding header. User defined properties are put in generic HTTP Headers which do not help. 

      My wish would be:

      1. FetchS3Object:
        1. reads the content-encoding and decodes gzip into non gzip with an attribute: Evaluate Content-Encoding 
        2. FetchS3 should create s3 header attributes which are stored in the s3 object 
      2. PutS3Object
        1. Has a Content-Encoding attribute where users can put "gzip" as a possible value
        2. Has an option to compress the flow file content in gzip with true|false. This would have a huge performance impact, because currently we need to compress the content, s3 upload and then decompress it afterwards 

       

      Thank you in advance

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            djakupovic Denis Jakupovic
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: