Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.9.0
-
None
-
None
-
Cloudera/CDH 6.1
Spark 2.4
Hadoop 3.0
zeppein-env.sh
export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3 export PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3 export PYSPARKPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3 export PYSPARKDRIVERPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
Cloudera/CDH 6.1 Spark 2.4 Hadoop 3.0 zeppein-env.sh export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3 export PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3 export PYSPARKPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3 export PYSPARKDRIVERPYTHON=/opt/cloudera/parcels/Anaconda/envs/py36/bin/python3
Description
Hi,
PySpark fails with following error in Zeppelin 0.9.0:
ERROR [2019-02-06 22:23:40,599] ({FIFOScheduler-Worker-1} Job.java[run]:174) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: Fail to bootstrap pyspark
at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:124)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:593)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:502)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:121)
at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Fail to run bootstrap script: python/zeppelin_pyspark.py
at org.apache.zeppelin.python.PythonInterpreter.bootstrapInterpreter(PythonInterpreter.java:581)
at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:122)
... 9 more
Full logs of YARN:
https://gist.github.com/maziyarpanahi/3380e230246271217a2feb4512f5d665