Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10298

Replaced dead node stayed in gossip forever

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.1.12, 2.2.4, 3.0.0
    • None
    • None
    • Normal

    Description

      The dead node stayed in the nodetool status,

      DN 10.210.165.55 379.76 GB 256 ? null

      And in the log, it throws NPE when trying to remove it.

      2015-09-10_06:41:22.92453 ERROR 06:41:22 Exception in thread Thread[GossipStage:1,5,main]
      2015-09-10_06:41:22.92454 java.lang.NullPointerException: null
      2015-09-10_06:41:22.92455       at org.apache.cassandra.utils.UUIDGen.decompose(UUIDGen.java:100) 
      2015-09-10_06:41:22.92455       at org.apache.cassandra.db.HintedHandOffManager.deleteHintsForEndpoint(HintedHandOffManager.java:201) 
      2015-09-10_06:41:22.92455       at org.apache.cassandra.service.StorageService.excise(StorageService.java:1886) 
      2015-09-10_06:41:22.92455       at org.apache.cassandra.service.StorageService.excise(StorageService.java:1902) 
      2015-09-10_06:41:22.92456       at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1805)
      2015-09-10_06:41:22.92457       at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1473) 
      2015-09-10_06:41:22.92457       at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2099) 
      2015-09-10_06:41:22.92457       at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1009) 
      2015-09-10_06:41:22.92458       at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1085) 
      2015-09-10_06:41:22.92458       at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) 
      2015-09-10_06:41:22.92458       at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) 
      2015-09-10_06:41:22.92459       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]
      2015-09-10_06:41:22.92460       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_45]
      

      Attachments

        1. CASSANDRA-10298.patch
          2 kB
          Dikang Gu

        Activity

          People

            dikanggu Dikang Gu
            dikanggu Dikang Gu
            Dikang Gu
            Stefania Alborghetti
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: