Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14136

MemtableFlushWriter DecoratedKey Exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Duplicate
    • None
    • Feature/2i Index, Legacy/Core
    • None

    • CentOS Linux release 7.3.1611 (Core) , 3 Node cluster with Replication Factor or 2

    • Normal

    Description

      Running into this issue on my cluster periodically for different tables. After this error is encountered, all the post flushes stop and eventually the system runs out of memory.
      On a restart all the commit logs get played normally and things go back to normal.

      I'm unable to understand the scenario, but the issue is recreating every few days.

      DEBUG [MemtableFlushWriter:884] 2017-12-26 18:19:40,883 Memtable.java:401 - Completed flushing /mnt/DATA/cassandra/data/products/products_by_hierarchy5storeid_pascdesc-411cabe0632411e7b25a1b665c06298b/.id
x_hierarchy1category/mc-2050-big-Data.db (508.127KiB) for commitlog position ReplayPosition(segmentId=1513929386900,
 position=19110822)
DEBUG [MemtableFlushWriter:884] 2017-12-26 18:19:41,150 Memtable.java:368 - Writing Memtable-products_by_hierarchy5storeid_pascdesc.idx_hierarchy3category@551487729(545.926KiB serialized bytes, 324073 ops, 0%/0% of on/off-heap limit)
ERROR [MemtableFlushWriter:884] 2017-12-26 18:19:41,316 CassandraDaemon.java:205 - Exception in thread Thread[MemtableFlushWriter:884,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(CU00328612, 43553030333238363132) >= current key DecoratedKey(^@^@^@^@^@^@^@^@^@^@, 43553030333238363838) writing into /mnt/DATA/cassandra/data/products/products_by_hierarchy5storeid_pascdesc-411cabe0632411e7b25a1b665c06298b/.idx_hierarchy3category/mc-2134-big-Data.db
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:394) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.db.Memtable.flush(Memtable.java:332) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]

      Attachments

        1. system.log.gz
          863 kB
          Aditya Bharadwaj
        2. debug.log.gz
          664 kB
          Aditya Bharadwaj

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. CASSANDRA-12125 ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137 CassandraDaemon.java:185 - Exception in thread Thread[MemtableFlushWriter:4,5,main] java.lang.RuntimeException: Last written key DecoratedKey(XXXX.XX, XXXXX) >= current key DecoratedKey

          • Normal -
          • Open

          Activity

            People

              Unassigned Unassigned
              adityabharadwaj Aditya Bharadwaj
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: