[ZEPPELIN-3656] Livy Code Completion does not work when using Shared Interpreter - ASF JIRA

Agile Board

Attach files

Attach Screenshot

Voters

Stop watching

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.8.0
Fix Version/s: 0.8.1, 0.9.0
Component/s: livy-interpreter
Labels:
None
Environment:

Zeppelin 0.8.0

Livy 0.5

Spark 2.1

Kerberos+SSL

Description

When attempting to use code completion with Livy 0.5 I get a NullPointerException:

2018-07-24 14:05:05,504 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.
java.lang.NullPointerException
at org.apache.zeppelin.livy.BaseLivyInterpreter.callCompletion(BaseLivyInterpreter.java:284)
at org.apache.zeppelin.livy.BaseLivyInterpreter.completion(BaseLivyInterpreter.java:271)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:138)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.completion(RemoteInterpreterServer.java:736)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1940)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1925)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-07-24 14:05:05,508 INFO org.apache.zeppelin.socket.NotebookServer: Fail to get completion
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:358)
at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:270)
at org.apache.zeppelin.notebook.Note.completion(Note.java:729)
at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:1397)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:303)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:372)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:355)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$8.call(RemoteInterpreter.java:362)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$8.call(RemoteInterpreter.java:359)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
... 21 more

Debugging this shows that it is the call to sessionInfo on this line: https://github.com/apache/zeppelin/blob/v0.8.0/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java#L284

The code should be referencing sessionInfo via the SessionInfo method to handle the case of shared interpreter.

It would be nice if this could be fixed in 0.8.1 and 0.9.