Major Description
The zeppelin server is consistently getting deadlocked when delete multiple paragraphs which belong to different notes synchronously
The RestApi called is below:
// zeppelin-server\src\main\java\org\apache\zeppelin\rest\NotebookRestApi.java /** Delete paragraph REST API * @param noteId ID of Note @return JSON with status.OK @throws IOException */ @DELETE @Path("{noteId}/paragraph/{paragraphId}") @ZeppelinApi public Response deleteParagraph(@PathParam("noteId") String noteId, @PathParam("paragraphId") String paragraphId) throws IOException { LOG.info("delete paragraph {} {}", noteId, paragraphId); Note note = notebook.getNote(noteId); checkIfNoteIsNotNull(note); checkIfUserCanRead(noteId, "Insufficient privileges you cannot remove paragraph from this note"); Paragraph p = note.getParagraph(paragraphId); checkIfParagraphIsNotNull(p); AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal()); note.removeParagraph(SecurityUtils.getPrincipal(), paragraphId); note.persist(subject); notebookServer.broadcastNote(note); return new JsonResponse(Status.OK, "").build(); }
The relevant thread dump is below:
Server compiler detected. JVM version is 25.212-b10 Deadlock Detection: Found one Java-level deadlock: ============================= "qtp1188392295-14051": waiting to lock Monitor@0x00007f3dc800a108 (Object@0x00000000c7706788, a java/util/LinkedList), which is held by "qtp1188392295-14087" "qtp1188392295-14087": waiting to lock Monitor@0x00007f3d10009278 (Object@0x00000000c78d3e78, a java/util/LinkedList), which is held by "qtp1188392295-14051" Found a total of 1 deadlock.
The stacktrace information is below:
"qtp1188392295-14087": [1] org.apache.zeppelin.notebook.Note.getParagraphs (Note.java:750) [2] org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime (Notebook.java:871) [3] org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo (NotebookServer.java:554) [4] org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove (NotebookServer.java:2,180) [5] org.apache.zeppelin.notebook.Notebook.onParagraphRemove (Notebook.java:1,083) [6] org.apache.zeppelin.notebook.Note.removeParagraph (Note.java:446) [7] org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph (NotebookRestApi.java:615) [8] sun.reflect.GeneratedMethodAccessor27.invoke (null) [9] sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) [10] java.lang.reflect.Method.invoke (Method.java:498) [11] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0 (ResourceMethodInvocationHandlerFactory.java:76) [12] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$159.invoke (null) [13] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run (AbstractJavaResourceMethodDispatcher.java:148) [14] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke (AbstractJavaResourceMethodDispatcher.java:191) [15] org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch (JavaResourceMethodDispatcherProvider.java:200) [16] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch (AbstractJavaResourceMethodDispatcher.java:103) [17] org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:493) [18] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:415) [19] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:104) [20] org.glassfish.jersey.server.ServerRuntime$1.run (ServerRuntime.java:277) [21] org.glassfish.jersey.internal.Errors$1.call (Errors.java:272) [22] org.glassfish.jersey.internal.Errors$1.call (Errors.java:268) [23] org.glassfish.jersey.internal.Errors.process (Errors.java:316) [24] org.glassfish.jersey.internal.Errors.process (Errors.java:298) [25] org.glassfish.jersey.internal.Errors.process (Errors.java:268) [26] org.glassfish.jersey.process.internal.RequestScope.runInScope (RequestScope.java:289) [27] org.glassfish.jersey.server.ServerRuntime.process (ServerRuntime.java:256) [28] org.glassfish.jersey.server.ApplicationHandler.handle (ApplicationHandler.java:703) [29] org.glassfish.jersey.servlet.WebComponent.serviceImpl (WebComponent.java:416) [30] org.glassfish.jersey.servlet.WebComponent.service (WebComponent.java:370) [31] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:389) [32] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:342) [33] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:229) [34] org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:865) [35] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,655) [36] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:61) [37] org.apache.shiro.web.servlet.AdviceFilter.executeChain (AdviceFilter.java:108) [38] org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal (AdviceFilter.java:137) [39] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125) [40] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:66) [41] org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain (AbstractShiroFilter.java:449) [42] org.apache.shiro.web.servlet.AbstractShiroFilter$1.call (AbstractShiroFilter.java:365) [43] org.apache.shiro.subject.support.SubjectCallable.doCall (SubjectCallable.java:90) [44] org.apache.shiro.subject.support.SubjectCallable.call (SubjectCallable.java:83) [45] org.apache.shiro.subject.support.DelegatingSubject.execute (DelegatingSubject.java:383) [46] org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal (AbstractShiroFilter.java:362) [47] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125) [48] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642) [49] org.apache.zeppelin.server.CorsFilter.doFilter (CorsFilter.java:72) [50] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642) [51] org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:533) [52] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:146) [53] org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:548) [54] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) [55] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257) [56] org.eclipse.jetty.server.session.SessionHandler.doHandle (SessionHandler.java:1,595) [57] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255) [58] org.eclipse.jetty.server.handler.ContextHandler.doHandle (ContextHandler.java:1,317) [59] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203) [60] org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473) [61] org.eclipse.jetty.server.session.SessionHandler.doScope (SessionHandler.java:1,564) [62] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201) [63] org.eclipse.jetty.server.handler.ContextHandler.doScope (ContextHandler.java:1,219) [64] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:144) [65] org.eclipse.jetty.server.handler.ContextHandlerCollection.handle (ContextHandlerCollection.java:219) [66] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) [67] org.eclipse.jetty.server.Server.handle (Server.java:531) [68] org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352) [69] org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:260) [70] org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:281) [71] org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102) [72] org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118) [73] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333) [74] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310) [75] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168) [76] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126) [77] org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366) [78] org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:762) [79] org.eclipse.jetty.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:680) [80] java.lang.Thread.run (Thread.java:748) ------------------------------------------------------------------------------------------------ "qtp1188392295-14051": [1] org.apache.zeppelin.notebook.Note.getParagraphs (Note.java:750) [2] org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime (Notebook.java:871) [3] org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo (NotebookServer.java:554) [4] org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove (NotebookServer.java:2,180) [5] org.apache.zeppelin.notebook.Notebook.onParagraphRemove (Notebook.java:1,083) [6] org.apache.zeppelin.notebook.Note.removeParagraph (Note.java:446) [7] org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph (NotebookRestApi.java:615) [8] sun.reflect.GeneratedMethodAccessor27.invoke (null) [9] sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) [10] java.lang.reflect.Method.invoke (Method.java:498) [11] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0 (ResourceMethodInvocationHandlerFactory.java:76) [12] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$159.invoke (null) [13] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run (AbstractJavaResourceMethodDispatcher.java:148) [14] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke (AbstractJavaResourceMethodDispatcher.java:191) [15] org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch (JavaResourceMethodDispatcherProvider.java:200) [16] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch (AbstractJavaResourceMethodDispatcher.java:103) [17] org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:493) [18] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:415) [19] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:104) [20] org.glassfish.jersey.server.ServerRuntime$1.run (ServerRuntime.java:277) [21] org.glassfish.jersey.internal.Errors$1.call (Errors.java:272) [22] org.glassfish.jersey.internal.Errors$1.call (Errors.java:268) [23] org.glassfish.jersey.internal.Errors.process (Errors.java:316) [24] org.glassfish.jersey.internal.Errors.process (Errors.java:298) [25] org.glassfish.jersey.internal.Errors.process (Errors.java:268) [26] org.glassfish.jersey.process.internal.RequestScope.runInScope (RequestScope.java:289) [27] org.glassfish.jersey.server.ServerRuntime.process (ServerRuntime.java:256) [28] org.glassfish.jersey.server.ApplicationHandler.handle (ApplicationHandler.java:703) [29] org.glassfish.jersey.servlet.WebComponent.serviceImpl (WebComponent.java:416) [30] org.glassfish.jersey.servlet.WebComponent.service (WebComponent.java:370) [31] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:389) [32] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:342) [33] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:229) [34] org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:865) [35] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,655) [36] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:61) [37] org.apache.shiro.web.servlet.AdviceFilter.executeChain (AdviceFilter.java:108) [38] org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal (AdviceFilter.java:137) [39] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125) [40] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:66) [41] org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain (AbstractShiroFilter.java:449) [42] org.apache.shiro.web.servlet.AbstractShiroFilter$1.call (AbstractShiroFilter.java:365) [43] org.apache.shiro.subject.support.SubjectCallable.doCall (SubjectCallable.java:90) [44] org.apache.shiro.subject.support.SubjectCallable.call (SubjectCallable.java:83) [45] org.apache.shiro.subject.support.DelegatingSubject.execute (DelegatingSubject.java:383) [46] org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal (AbstractShiroFilter.java:362) [47] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125) [48] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642) [49] org.apache.zeppelin.server.CorsFilter.doFilter (CorsFilter.java:72) [50] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642) [51] org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:533) [52] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:146) [53] org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:548) [54] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) [55] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257) [56] org.eclipse.jetty.server.session.SessionHandler.doHandle (SessionHandler.java:1,595) [57] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255) [58] org.eclipse.jetty.server.handler.ContextHandler.doHandle (ContextHandler.java:1,317) [59] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203) [60] org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473) [61] org.eclipse.jetty.server.session.SessionHandler.doScope (SessionHandler.java:1,564) [62] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201) [63] org.eclipse.jetty.server.handler.ContextHandler.doScope (ContextHandler.java:1,219) [64] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:144) [65] org.eclipse.jetty.server.handler.ContextHandlerCollection.handle (ContextHandlerCollection.java:219) [66] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) [67] org.eclipse.jetty.server.Server.handle (Server.java:531) [68] org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352) [69] org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:260) [70] org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:281) [71] org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102) [72] org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118) [73] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333) [74] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310) [75] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168) [76] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126) [77] org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366) [78] org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:762) [79] org.eclipse.jetty.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:680) [80] java.lang.Thread.run (Thread.java:748)