Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.8.0
-
None
-
None
Description
We have encountered a failure of TimeoutLifecycleManagement before, and combined the relevant code according to ZEPPELIN-4133, but the problem still appears.I upload full log
I found that the related process of TimeoutLifecycleManagement was blocked,and Waiting for 0x00000002df8365c0 lock
"pool-3-thread-1" #33 prio=5 os_prio=0 tid=0x00007f86d970f800 nid=0x29ef waiting for monitor entry [0x00007f84ff42c000]"pool-3-thread-1" #33 prio=5 os_prio=0 tid=0x00007f86d970f800 nid=0x29ef waiting for monitor entry [0x00007f84ff42c000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.close(ManagedInterpreterGroup.java:89) - waiting to lock <0x00000002df8365c0> (a org.apache.zeppelin.interpreter.ManagedInterpreterGroup) at org.apache.zeppelin.interpreter.lifecycle.TimeoutLifecycleManager.lambda$new$0(TimeoutLifecycleManager.java:50) at org.apache.zeppelin.interpreter.lifecycle.TimeoutLifecycleManager$$Lambda$64/1972143064.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
thread with 0x00000002df8365c0 lock is doing some close interpreter actions, but it seems to be "blocked", causing 0x00000002df8365c0 lock to never be released.
"qtp1659791576-43255" #43255 prio=5 os_prio=0 tid=0x00007f8170124800 nid=0x394d runnable [0x00007f816e8e6000]"qtp1659791576-43255" #43255 prio=5 os_prio=0 tid=0x00007f8170124800 nid=0x394d runnable [0x00007f816e8e6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000002df85e828> (a java.io.BufferedInputStream) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_shutdown(RemoteInterpreterService.java:394) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.shutdown(RemoteInterpreterService.java:382) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess$4.call(RemoteInterpreterManagedProcess.java:240) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess$4.call(RemoteInterpreterManagedProcess.java:237) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.stop(RemoteInterpreterManagedProcess.java:237) at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.close(ManagedInterpreterGroup.java:109) - locked <0x00000002df8365c0> (a org.apache.zeppelin.interpreter.ManagedInterpreterGroup) at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.close(ManagedInterpreterGroup.java:91) - locked <0x00000002df8365c0> (a org.apache.zeppelin.interpreter.ManagedInterpreterGroup) at org.apache.zeppelin.interpreter.InterpreterSetting.close(InterpreterSetting.java:485) at org.apache.zeppelin.interpreter.InterpreterSettingManager.close(InterpreterSettingManager.java:979) at org.apache.zeppelin.rest.InterpreterRestApi.restartSetting(InterpreterRestApi.java:249) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$132/1114616259.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoke