Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14839

Cannot bootstrap new node IOException: CF <GUID> was dropped during streaming - Cassandra 3.0.16

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • Cassandra 3.0.16

    • Normal

    Description

      I have a Cassandra 3.0.16 cluster with 6 nodes. 

      Repairs work well on all nodes. 

      Trying to join a 7th node fails with:

      ERROR [STREAM-IN-/172.30.2.191] 2018-10-23 20:57:06,497 StreamSession.java:534 - [Stream #08d98990-d706-11e8-bf10-9b02b0f15309] Streaming error occurred
      java.io.IOException: CF 9428b300-99ac-11e8-aa56-b1d7e983933e was dropped during streaming
      at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:54) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:43) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:294) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.16.jar:3.0.16]
      at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
      

      On disk I have:

      ubuntu@prd-cassandra-2d:/data$ find -type d -name '*9428b30099ac11e8aa56b1d7e983933e*'
      ./cassandra/data/sandboxprdvault/vault-9428b30099ac11e8aa56b1d7e983933e
      

      However in system_schema.tables I have:

      keyspace_name	table_name	id
      sandboxprdvault	vault	942c8390-99ac-11e8-be55-69bc3438b9fa
      

      What is unusual to me is that the system_schema.table.ID for vault is different than the id on disk for the same table. 

      This seems to cause a stream error when bootstrapping a new node. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            scubadrew Drew O'Connor
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: