Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32039

Unable to Set `spark.ui.port` configuration in Yarn Cluster Mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 2.4.0
    • None
    • Web UI
    • None

    Description

      Spark Web UI port in Yarn cluster mode always gets a random number since we disable the configuration spark.ui.port by setting this property to be 0 always. 

      There are some use cases where we need to run Web UI port on some specified range because of some security concerns and unable to do so with the configuration. 

       

      https://github.com/apache/spark/blob/842b1dcdff0ecab4af9f292c2ff7b2b9ae1ac40a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L216

      Attachments

        Activity

          People

            Unassigned Unassigned
            ahujarajat261 rajat
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified