Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8619

A new replica should not become leader when all current replicas are down as it leads to data loss

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 5.5
    • None
    • None

    Description

      Here's what I'm talking about:

      • Start a 2 node solrcloud cluster
      • Create a 1 shard/1 replica collection
      • Add documents
      • Shut down the node that has the only active shard
      • ADDREPLICA for the shard/collection, so Solr would attempt to add a new replica on the other node
      • Solr waits for a while before this replica becomes an active leader.
      • Index a few new docs
      • Bring up the old node
      • The replica comes up, with it's old index and then syncs to only contain the docs from the new leader.
        All old documents are lost in this case

      Here are a few things that might work here:
      1. Reject an ADDREPLICA call if all current replicas for the shard are down. Considering the new replica can not sync from anyone, it doesn't make sense for this replica to even come up
      2. The replica shouldn't become active/leader unless either it was the last known leader or active before it went into recovering state
      unless there are no other replicas in the clusterstate.

      This might very well be related to SOLR-8173 but we should add a check to ADDREPLICA as well.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            anshum Anshum Gupta
            anshum Anshum Gupta

            Dates

              Created:
              Updated:

              Slack

                Issue deployment