Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32035

Inconsistent AWS environment variables in documentation

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.6, 3.0.0
    • 2.4.7, 3.0.1, 3.1.0
    • Documentation
    • None

    Description

      Looking at the actual Scala code, the environment variables used to log into AWS are:

      • AWS_ACCESS_KEY_ID
      • AWS_SECRET_ACCESS_KEY
      • AWS_SESSION_TOKEN

      These are the same that AWS uses in their libraries.

      However, looking through the Spark documentation and comments, I see that these are not denoted correctly across the board:

      docs/cloud-integration.md
      106:1. `spark-submit` reads the `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` <-- both different
      107:and `AWS_SESSION_TOKEN` environment variables and sets the associated authentication options

      docs/streaming-kinesis-integration.md
      232:- Set up the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_KEY` with your AWS credentials. <-- secret key different

      external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
      34: $ export AWS_ACCESS_KEY_ID=<your-access-key>
      35: $ export AWS_SECRET_KEY=<your-secret-key> <-- different
      48: Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY <-- secret key different

      core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
      438: val keyId = System.getenv("AWS_ACCESS_KEY_ID")
      439: val accessKey = System.getenv("AWS_SECRET_ACCESS_KEY")
      448: val sessionToken = System.getenv("AWS_SESSION_TOKEN")

      external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
      53: * $ export AWS_ACCESS_KEY_ID=<your-access-key>
      54: * $ export AWS_SECRET_KEY=<your-secret-key> <-- different
      65: * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY <-- secret key different

      external/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java
      59: * $ export AWS_ACCESS_KEY_ID=[your-access-key]
      60: * $ export AWS_SECRET_KEY=<your-secret-key> <-- different
      71: * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY <-- secret key different

      Attachments

        Activity

          People

            rjoerger Richard Joerger
            ondrej Ondrej Kokes
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: