Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14336

sstableloader fails if sstables contains removed columns

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • Legacy/Tools
    • None
    • Normal

    Description

      If I copy the schema and try to load in sstables with sstableloader, loading sometimes fails with

      Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.lang.RuntimeException: Failed to list files in /tmp/test/bug3_dest-acdc
          at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:93)
          at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
      Caused by: java.lang.RuntimeException: Failed to list files in /tmp/test/bug3_dest-acdc
          at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
          at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:561)
          at org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:76)
          at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:165)
          at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:80)
          ... 1 more
      Caused by: java.lang.RuntimeException: Unknown column d during deserialization
          at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:321)
          at org.apache.cassandra.io.sstable.format.SSTableReader.openForBatch(SSTableReader.java:440)
          at org.apache.cassandra.io.sstable.SSTableLoader.lambda$openSSTables$0(SSTableLoader.java:121)
          at org.apache.cassandra.db.lifecycle.LogAwareFileLister.lambda$innerList$2(LogAwareFileLister.java:99)
          at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
          at java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
          at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
          at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
          at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
          at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
          at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
          at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:101)
          at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)
          ... 5 more

      This requires that we have dropped columns in the source table and sstables exist from the "old schema" time.

      This can be very easily reproduced. I used following script:

      KS=test
      SRCTABLE=bug3_source
      DESTTABLE=bug3_dest
      DATADIR=/usr/local/var/lib/cassandra/data
      TMPDIR=/tmp
      
      cqlsh -e "CREATE TABLE $KS.$SRCTABLE(a int primary key, b int, c int, d int);"
      cqlsh -e "CREATE TABLE $KS.$DESTTABLE(a int primary key, b int, c int);"
      cqlsh -e "INSERT INTO $KS.$SRCTABLE(a,b,c,d) values(1,2,3,4);"
      nodetool flush $KS $SRCTABLE
      cqlsh -e "ALTER TABLE $KS.$SRCTABLE DROP d;"
      nodetool flush $KS $SRCTABLE
      
      mkdir -p $TMPDIR/$KS/$DESTTABLE-acdc
      cp $DATADIR/$KS/$SRCTABLE-*/* $TMPDIR/$KS/$DESTTABLE-acdc
      sstableloader -d 127.0.0.1 $TMPDIR/$KS/$DESTTABLE-acdc

      Attachments

        Issue Links

          Activity

            People

              chovatia.jaydeep@gmail.com Jaydeepkumar Chovatia
              hkroger Hannu Kröger
              Jaydeepkumar Chovatia
              Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: