Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37005

pyspark os.getenv('SPARK_YARN_STAGING_DIR') can not get job path

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.4.4
    • None
    • PySpark
    • python2.7

      spark2.4

    Description

      when  i submit a spark job , use spark-submit and set option --files file1, 

      then in python code, i use 

      //代码占位符
      path = str(os.environ["SPARK_YARN_STAGING_DIR"])
      

      return path is None, but this can success in java code

      //代码占位符
      spark.read().textFile(System.getenv("SPARK_YARN_STAGING_DIR") + "/README.md")
      

      which cause this problem.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            ZhaoWeiNan WeiNan Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: