Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26606

parameters passed in extraJavaOptions are not being picked up

    XMLWordPrintableJSON

Details

    Description

      driver.extraJavaOptions and executor.extraJavaOptions are not being picked up . Even though I see the parameters are being passed fine in the spark launch command I do not see these parameters are being picked up for some unknown reason. My source code throws an error stating the java params are empty

       

      This is my spark submit command: 

          output=`spark-submit \
      --class com.demo.myApp.App \
      --conf 'spark.executor.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=file:/dev/./urandom' \
      --conf 'spark.driver.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC -Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=file:/dev/./urandom' \
      --executor-memory "$EXECUTOR_MEMORY" \
      --executor-cores "$EXECUTOR_CORES" \
      --total-executor-cores "$TOTAL_CORES" \
      --driver-memory "$DRIVER_MEMORY" \
      --deploy-mode cluster \
      /home/spark/asm//current/myapp-*.jar 2>&1 &`

       

       

      Is there any other way I can access the java params with out using extraJavaOptions. 

      Attachments

        Activity

          People

            kabhwan Jungtaek Lim
            rrb441 Ravindra
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: