Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-17674

Test failure: org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • 4.0.x, 4.1.x
    • Test/dtest/java
    • None
    • Correctness - Test Failure
    • Normal
    • Normal
    • User Report
    • All
    • None
    • Hide

      See comments

      Show
      See comments

    Description

      The Java upgrade dtest org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage is ~68% flaky on 4.0 and ~2% flaky on trunk, at least in CircleCI:

      The error for 4.0 is:

      java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
      	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
      	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:227)
      	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
      Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:235)
      	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:231)
      Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:62)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$28(Instance.java:810)
      	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$null$8(IsolatedExecutor.java:114)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      Caused by: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
      	at org.apache.cassandra.utils.ExecutorUtils.awaitTerminationUntil(ExecutorUtils.java:107)
      	at org.apache.cassandra.utils.ExecutorUtils.awaitTermination(ExecutorUtils.java:96)
      	at org.apache.cassandra.utils.ExecutorUtils.shutdownNowAndWait(ExecutorUtils.java:139)
      	at org.apache.cassandra.concurrent.StageManager.shutdownAndWait(StageManager.java:128)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$null$26(Instance.java:803)
      	at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$32(Instance.java:968)
      

      Whereas the error for trunk is:

      org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test
      	at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056)
      	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042)
      	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:244)
      	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
      	Suppressed: java.lang.NullPointerException
      		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:992)
      		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:987)
      		at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2512)
      		at org.apache.cassandra.service.StorageService.onChange(StorageService.java:2086)
      		at org.apache.cassandra.gms.Gossiper.markAsShutdown(Gossiper.java:479)
      		at org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37)
      		at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      		at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
      		at java.lang.Thread.run(Thread.java:748)
      

      I haven't managed to reproduce it on 4.1, although more iterations could do it.

      This was discovered during CASSANDRA-17213.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bereng Berenguer Blasi Assign to me
            adelapena Andres de la Peña
            Berenguer Blasi

            Dates

              Created:
              Updated:

              Slack

                Issue deployment