Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10579

IndexOutOfBoundsException during memtable flushing at startup (with offheap_objects)

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.1.12, 2.2.4
    • None
    • None

    • 2.1.10 on linux

    • Normal

    Description

      Sometimes we have problems at startup where memtable flushes with an index out of bounds exception as seen below. Cassandra is then dead in the water until we track down the corresponding commit log via the segment ID and remove it:

      INFO  [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
INFO  [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log (CL version 4, messaging version 8)
INFO  [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
INFO  [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
INFO  [main] 2015-10-23 14:43:36,595 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log (CL version 4, messaging version 8)
INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log
INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log (CL version 4, messaging version 8)
WARN  [SharedPool-Worker-5] 2015-10-23 14:43:36,747 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-5,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: 6
        at org.apache.cassandra.db.AbstractNativeCell.nametype(AbstractNativeCell.java:204) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.AbstractNativeCell.isStatic(AbstractNativeCell.java:199) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.composites.AbstractCType.compare(AbstractCType.java:166) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:61) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:58) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.BTree.find(BTree.java:277) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:154) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.Builder.update(Builder.java:74) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.BTree.update(BTree.java:186) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:225) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1225) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:455) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_31]
        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-10626 Seeing repeated Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.AssertionError: null in logs

          • Normal -
          • Resolved

          Activity

            People

              benedict Benedict Elliott Smith
              jeffery.griffith Jeff Griffith
              Benedict Elliott Smith
              Branimir Lambov
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: