Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13044

saveAsTextFile(s3n://) doesn't support s3 Signature Version 4

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 1.4.0
    • None
    • Input/Output

    • CentOS

    Description

      I have two clusters deployed: US and EU-Frankfort with the same configs on AWS.

      And the application in EU-Frankfort cannot save data to EU-Frankfort-s3, but US one can save to US-s3.

      And I checked and found that EU-Frankfort supports Signature Version 4 only: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

      Code I'm using:

      val s3WriteEndpoint = "s3n://access_key:secret_key@bucket_name/data/12345"
rdd.saveAsTextFile(s3WriteEndpoint)

      So from my issue I guess saveAsTextFile() is using Signature Version 2? How to support Version 4?

      I tried to dig into code
      https://github.com/apache/spark/blob/f14922cff84b1e0984ba4597d764615184126bdc/core/src/main/scala/org/apache/spark/rdd/RDD.scala

      Attachments

        Issue Links

          is broken by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13325 s3n fails to work with S3 Frankfurt or Seoul - 400 : Bad Request

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-11353 Writing to S3 buckets, which only support AWS4-HMAC-SHA256 fails with s3n URLs

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              iamshrek Xin Ren
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: