Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29202

--driver-java-options are not passed to driver process in yarn client mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Deploy
    • None

    Description

      Run the below command 

      ./bin/spark-sql --master yarn --driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"

       
      In Spark 2.3.3

      /opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ -Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal

       
      In Spark 3.0

      /opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal

      We can see that java options are not passed to driver process in spark3

       

      Attachments

        Issue Links

          Activity

            People

              sandeep.katta2007 Sandeep Katta
              sandeep.katta2007 Sandeep Katta
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: