Improve cache block migration

We should explore the following improvements to cache block migration:

1) Peer selection (right now may overbalance on certain peers)

2) Do we need to configure the number of blocks to be migrated at the same time

3) Are there any blocks we don't need to replicate (e.g. they are already stored on the desired number of executors even once we remove the executors slated for decommissioning).

4) Do we want to prioritize migrating blocks with no replicas

5) Log the attempt number for debugging 

6) Clarify the logic for determining the number of replicas

7) Consider using TestUtils.waitUntilExecutorsUp in tests rather than count to wait for the executors to come up. imho this is the least important.