Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.6.0
-
None
Description
We run zeppelin on spark when spark authentication is turned on, but got the following exception
INFO [2016-07-28 00:35:32,845] ({pool-2-thread-2} Logging.scala[logInfo]:58) - Changing view acls to: zeppelin
INFO [2016-07-28 00:35:32,846] ({pool-2-thread-2} Logging.scala[logInfo]:58) - Changing modify acls to: zeppelin
INFO [2016-07-28 00:35:32,908] ({pool-1-thread-3} Logging.scala[logInfo]:58) - Changing view acls to: zeppelin
INFO [2016-07-28 00:35:32,908] ({pool-1-thread-3} Logging.scala[logInfo]:58) - Changing modify acls to: zeppelin
ERROR [2016-07-28 00:35:32,909] ({pool-2-thread-2} Job.java[run]:189) - Job failed
java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:397)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:219)
at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:217)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:566)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR [2016-07-28 00:35:32,918] ({pool-1-thread-3} TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:397)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:219)
at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:217)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:566)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:110)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:404)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1509)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1494)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The root cause is that we didn't set property SPARK_YARN_MODE just like spark-shell did.
The following code in Main.scala is what we miss in zeppelin spark interpreter. https://github.com/apache/spark/blob/branch-1.6/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
def main(args: Array[String]) { if (getMaster == "yarn-client") System.setProperty("SPARK_YARN_MODE", "true")
Thanks kbadani for finding this issue.
Attachments
Issue Links
- blocks
-
ZEPPELIN-1177 Release 0.6.1
-
- Resolved
-
- links to
