Description
Run the below command
./bin/spark-sql --master yarn --driver-java-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5555"
In Spark 2.3.3
/opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/conf/:/opt/BigdataTools/spark-2.3.3-bin-hadoop2.7/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ -Xmx1g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5555 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
In Spark 3.0
/opt/softwares/Java/jdk1.8.0_211/bin/java -cp /opt/apache/git/sparkSourceCode/spark/conf/:/opt/apache/git/sparkSourceCode/spark/assembly/target/scala-2.12/jars/*:/opt/BigdataTools/hadoop-3.2.0/etc/hadoop/ org.apache.spark.deploy.SparkSubmit --master yarn --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5556 --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver spark-internal
We can see that java options are not passed to driver process in spark3
Attachments
Issue Links
- links to