Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-10758

Avoid or mitigate Cursor Timeout in Mongo when running detailedGC

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • documentmk, oak-run
    • None

    Description

      When running oak-run revisions detailedGC command, if the cleanup is too big the processing time can exceed the Mongo cursor timeout, which defaults to 30 minutes:

      java.io.IOException: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1007212049743349339 not found on server mongohost' on server mongohost:27017
          at org.apache.jackrabbit.oak.run.RevisionsCommand.collectOnce(RevisionsCommand.java:427)
          at org.apache.jackrabbit.oak.run.RevisionsCommand.collect(RevisionsCommand.java:377)
          at org.apache.jackrabbit.oak.run.RevisionsCommand.execute(RevisionsCommand.java:245)
          at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
      Caused by: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1007212049743349339 not found on server mongohost:27017' on server mongohost:27017
          at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27)
          at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:268)
          at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:139)
          at com.mongodb.client.internal.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:54)
          at org.apache.jackrabbit.guava.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
          at org.apache.jackrabbit.guava.common.collect.Iterators$5.computeNext(Iterators.java:670)
          at org.apache.jackrabbit.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
          at org.apache.jackrabbit.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
          at org.apache.jackrabbit.guava.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1395)
          at org.apache.jackrabbit.oak.plugins.document.SplitDocumentCleanUp.disconnect(SplitDocumentCleanUp.java:63)
          at org.apache.jackrabbit.oak.plugins.document.VersionGCSupport.deleteSplitDocuments(VersionGCSupport.java:146)
          at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.collectSplitDocuments(VersionGarbageCollector.java:842)
          at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.gc(VersionGarbageCollector.java:686)
          at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.run(VersionGarbageCollector.java:653)
          at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector.gc(VersionGarbageCollector.java:221)
          at org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:401)
          at org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:398)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:834)
      15:45:08.184 [DocumentNodeStore cluster update thread (0)] WARN  o.a.j.o.p.document.DocumentNodeStore - Background operation failed: java.lang.reflect.UndeclaredThrowableException
      java.lang.reflect.UndeclaredThrowableException: null
          at com.sun.proxy.$Proxy0.query(Unknown Source)
          at org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfoDocument.all(ClusterNodeInfoDocument.java:154)
          at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.updateClusterState(DocumentNodeStore.java:2480)
          at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundClusterUpdate.execute(DocumentNodeStore.java:3931)
          at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3734)
          at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.lang.reflect.InvocationTargetException: null
          at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:566)
          at org.apache.jackrabbit.oak.plugins.document.util.ReadOnlyDocumentStoreWrapperFactory$1.invoke(ReadOnlyDocumentStoreWrapperFactory.java:41)
          ... 6 common frames omitted
      Caused by: java.lang.IllegalStateException: The pool is closed
          at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:137)
          at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:123)
          at com.mongodb.internal.session.ServerSessionPool.get(ServerSessionPool.java:80)
          at com.mongodb.internal.session.BaseClientSessionImpl.<init>(BaseClientSessionImpl.java:44)
          at com.mongodb.client.internal.ClientSessionImpl.<init>(ClientSessionImpl.java:56)
          at com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:120)
          at com.mongodb.Mongo.createClientSession(Mongo.java:837)
          at com.mongodb.MongoClient.startSession(MongoClient.java:582)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoSessionFactory.createClientSession(MongoSessionFactory.java:55)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDBConnection.createClientSession(MongoDBConnection.java:115)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createClientSession(MongoDocumentStore.java:2405)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.execute(MongoDocumentStore.java:2389)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryInternal(MongoDocumentStore.java:846)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryWithRetry(MongoDocumentStore.java:774)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:751)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:739)
          at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:728)
          ... 10 common frames omitted 

      Check if this needs mitigation or could happen as part of the internal repository detailedGC execution.

      Attachments

        Issue Links

          Activity

            People

              corderob José Andrés Cordero Benítez
              corderob José Andrés Cordero Benítez
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: