Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6662

Allow variable substitution in spark.yarn.historyServer.address

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.3.0
    • 1.4.0
    • YARN

    Description

      In Spark on YARN, explicit hostname and port number need to be set for "spark.yarn.historyServer.address" in SparkConf to make the HISTORY link. If the history server address is known and static, this is usually not a problem.

      But in cloud, that is usually not true. Particularly in EMR, the history server always runs on the same node as with RM. So I could simply set it to ${yarn.resourcemanager.hostname}:18080 if variable substitution is allowed.

      In fact, Hadoop configuration already implements variable substitution, so if this property is read via YarnConf, this can be easily achievable.

      Attachments

        Activity

          People

            cheolsoo Cheolsoo Park
            cheolsoo Cheolsoo Park
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: