Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-3656

Livy Code Completion does not work when using Shared Interpreter

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.8.0
    • 0.8.1, 0.9.0
    • livy-interpreter
    • None
    • Zeppelin 0.8.0

      Livy 0.5

      Spark 2.1

      Kerberos+SSL

    Description

      When attempting to use code completion with Livy 0.5 I get a NullPointerException:

       

      2018-07-24 14:05:05,504 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.
      java.lang.NullPointerException
      at org.apache.zeppelin.livy.BaseLivyInterpreter.callCompletion(BaseLivyInterpreter.java:284)
      at org.apache.zeppelin.livy.BaseLivyInterpreter.completion(BaseLivyInterpreter.java:271)
      at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:138)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.completion(RemoteInterpreterServer.java:736)
      at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1940)
      at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$completion.getResult(RemoteInterpreterService.java:1925)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      2018-07-24 14:05:05,508 INFO org.apache.zeppelin.socket.NotebookServer: Fail to get completion
      java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:358)
      at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:270)
      at org.apache.zeppelin.notebook.Note.completion(Note.java:729)
      at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:1397)
      at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:303)
      at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
      at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
      at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
      at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
      at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
      at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
      at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
      at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
      at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
      at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
      at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
      at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
      at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.thrift.transport.TTransportException
      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
      at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:372)
      at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:355)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$8.call(RemoteInterpreter.java:362)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$8.call(RemoteInterpreter.java:359)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
      ... 21 more
      

      Debugging this shows that it is the call to sessionInfo on this line: https://github.com/apache/zeppelin/blob/v0.8.0/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java#L284

       

      The code should be referencing sessionInfo via the SessionInfo method to handle the case of shared interpreter.

       

      It would be nice if this could be fixed in 0.8.1 and 0.9.

       

      Attachments

        Issue Links

          Activity

            People

              abush Alex Bush
              bushnoh Alex Bush
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: