autoAddReplicas is not reliable when multiple nodes go down.

I started noticing problems in our production environment with indexing being blocked due to a minimum replication factor not being met.  We have autoAddReplicas triggers in place to add replicas when nodes our lost but it doesn't seem to correctly add all replicas that have been lost when nodes are lost. I’ve been able to reproduce this behavior consistently in a development environment.

Repro:

1. Setup a 10 node SolrCloud cluster.
2. Create autoAddReplicas to trigger on nodeLost with waitFor set to 10 minutes.
3. Create 15 collections with 2 shards and 4 replicas.
4. Kill 3 Solr nodes.
5. 15 minutes later kill 1 more Solr node.

Results:

Monitor your shards/replicas.  You’ll see some replicas added to make up for the lost replicas but not all.  An hour later many shards are still missing replicas.

Expected:

All lost replicas should be added on the 6 remaining healthy nodes.