Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10157

Add ability to specify s3 bootstrap script to spark-ec2

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • None
    • None
    • EC2

    Description

      One of the nice features of using EMR to start spark clusters is that you can specify bootstrap actions to run on each node. This is useful for installing custom libraries, etc. However, you pay a per-node/per-hour premium for using EMR (https://aws.amazon.com/elasticmapreduce/pricing/).

      This pull request adds the --bootstrap-script flag to the spark-ec2 script. This flag specifies an s3 path to a shell script. The script is downloaded and run on each node during the setup process. The flag can be specified multiple times for multiple bootstrap actions.

      Attachments

        Activity

          People

            Unassigned Unassigned
            mdagost Michelangelo D'Agostino
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: