Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
1.8.22
-
None
-
None
-
Oak: 1.8.22
Mongo Node Store
S3 Blob Store
Description
When attempting to save a new item of content into Oak, we are now getting the following message. As mentioned in a previous issue raised (OAK-9112), we have recently backloaded a lot of content into Oak with the Mongo "nodes" collection increasing from 100,000 documents to 600,000 documents.
The new content we are creating is less than 1k in size.
I appreciate this probably not a bug but any guidance resolving this issue in a production system would be appreciated.
We can update existing content items without issue only creating new items is causing the stack trace below.
16 Jun 2020 08:03:34.263HostName=crms-service-content-f899d5f6d-lwmgp 08:03:34.223 [content-service] [http-nio-8080-exec-195] ERROR o.s.c.s.instrument.web.TraceFilter - Uncaught exception thrown org.springframework.web.util.NestedServletException: Request processing failed; nested exception is javax.jcr.InvalidItemStateException: OakMerge0001: OakMerge0001: Failed to merge changes to the underlying store (retries 5, 70255 ms)
15 Jun 2020 19:07:33.228HostName=crms-service-content-f899d5f6d-qr9lt 19:07:33.126 [content-service] [http-nio-8080-exec-311] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is javax.jcr.InvalidItemStateException: OakMerge0001: OakMerge0001: Failed to merge changes to the underlying store (retries 5, 68221 ms)] with root cause com.mongodb.MongoCommandException: Command failed with error 17419: 'Resulting document after update is larger than 16777216' on server sg-pbprod-10448.servers.mongodirector.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Resulting document after update is larger than 16777216", "code" : 17419, "codeName" : "Location17419" } at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115) at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114) at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168) at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289) at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216) at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:153) at com.mongodb.operation.FindAndUpdateOperation$1.call(FindAndUpdateOperation.java:335) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:426) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:417) at com.mongodb.operation.FindAndUpdateOperation.execute(FindAndUpdateOperation.java:331) at com.mongodb.Mongo.execute(Mongo.java:845) at com.mongodb.Mongo$2.execute(Mongo.java:828) at com.mongodb.DBCollection.findAndModify(DBCollection.java:1928) at com.mongodb.DBCollection.findAndModify(DBCollection.java:1792) at com.mongodb.DBCollection.findAndModify(DBCollection.java:1761) at com.mongodb.DBCollection.findAndModify(DBCollection.java:1708) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.findAndModify(MongoDocumentStore.java:862) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createOrUpdate(MongoDocumentStore.java:908) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createOrUpdate(MongoDocumentStore.java:987) at org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.createOrUpdate(LeaseCheckDocumentStoreWrapper.java:136) at org.apache.jackrabbit.oak.plugins.document.Commit.applyToDocumentStore(Commit.java:343) at org.apache.jackrabbit.oak.plugins.document.Commit.applyToDocumentStore(Commit.java:264) at org.apache.jackrabbit.oak.plugins.document.Commit.applyInternal(Commit.java:233) at org.apache.jackrabbit.oak.plugins.document.Commit.apply(Commit.java:222) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.persist(DocumentNodeStoreBranch.java:301) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.persist(DocumentNodeStoreBranch.java:271) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.access$400(DocumentNodeStoreBranch.java:57) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch$InMemory.merge(DocumentNodeStoreBranch.java:519) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.merge0(DocumentNodeStoreBranch.java:188) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.merge(DocumentNodeStoreBranch.java:130) at org.apache.jackrabbit.oak.plugins.document.DocumentRootBuilder.merge(DocumentRootBuilder.java:163) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.merge(DocumentNodeStore.java:1914) at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:250) at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:346) at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.save(SessionDelegate.java:493) at org.apache.jackrabbit.oak.jcr.session.SessionImpl$8.performVoid(SessionImpl.java:424) at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:273) at org.apache.jackrabbit.oak.jcr.session.SessionImpl.save(SessionImpl.java:421) at au.com.calibreft.service.content.dao.JcrRepository.executeWithSession(JcrRepository.java:527) at au.com.calibreft.service.content.dao.JcrRepository.executeSaveWithSession(JcrRepository.java:494) at au.com.calibreft.service.content.dao.JcrRepository.save(JcrRepository.java:472) at au.com.calibreft.service.content.dao.JcrRepository$$FastClassBySpringCGLIB$$b81905ec.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671) at au.com.calibreft.service.content.dao.JcrRepository$$EnhancerBySpringCGLIB$$a5620910.save(<generated>) at au.com.calibreft.service.content.rest.ContentController.saveContent(ContentController.java:126) at sun.reflect.GeneratedMethodAccessor286.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:54) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)