Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
  3. HADOOP-19033

S3A: disable checksums when fs.s3a.checksum.validation = false

    XMLWordPrintableJSON

Details

    Description

      AWS v2 sdk turns on client-side checksum validation; this kills performance

      Given we are using TLS to download from AWS s3, there's implicit channel checksumming going on on, that's along with the IPv4 TCP checksumming.

      We don't need it, all it does is slow us down.

      proposed: disable in DefaultS3ClientFactory

      I don't want to add an option to enable it as it only complicates life (yet another config option), but I am open to persuasion

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: