Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2116

livy interpreter needs to restart after previous session timeout in secured cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.7.1, 0.8.0
    • livy-interpreter
    • None

    Description

      1) Reduce livy.session.timeout = 300000 (5 min)
      2) I started a long running job
      %livy
      print(1)
      Thread.sleep(400000)
      print(2)

      This scenario runs fine.
      3) After a period of inactivity, the livy session times out. I can see that on RM UI - the livy session state is FINISHED
      4) Now in the same notebook, if I try to do
      %livy
      sc.version
      It gives 404 Error

      org.springframework.web.client.HttpClientErrorException: 404 Not Found
      at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
      at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667)
      at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620)
      at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202)
      at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67)
      at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:360)
      at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)
      at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
      at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
      at org.apache.zeppelin.livy.BaseLivyInterprereter.callRestAPI(BaseLivyInterprereter.java:398)
      at org.apache.zeppelin.livy.BaseLivyInterprereter.executeStatement(BaseLivyInterprereter.java:359)
      at org.apache.zeppelin.livy.BaseLivyInterprereter.interpret(BaseLivyInterprereter.java:229)
      at org.apache.zeppelin.livy.BaseLivyInterprereter.interpret(BaseLivyInterprereter.java:140)
      at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:495)
      at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
      at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      5) Now restart the livy interpreter, and if you again try to do sc.version - it works

      Attachments

        Issue Links

          Activity

            People

              zjffdu Jeff Zhang
              kbadani Kshitij Badani
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: