Limit pendingReconstruction size

Consider the following scenario：

 (1) The hdfs cluster is big enough. 

Namenode Config： 

DFS_NAMENODE_REPLICATION_WORK_MULTIPLIER_PER_ITERATION ->200

DFS_NAMENODE_REPLICATION_MAX_STREAMS_KEY->100

DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_KEY->200

Datanode ：

It has large disk on one datanode.  One datanode has millions or even tens of millions blocks.  Or maybe more in archive cluster.

(2) The cluster administrator may do one of the following actions：

2.1 Increase replication on one large dir ,eg 2PB 

2.2 Decommissioning  some nodes from cluster

(3)  When the cluster datanode is on heavy read and write.

Thus, datanode receive transfer commands form heartbeat whit namenode. When datanode is on heavily  load （write and read）, the SumOfActorCommandQueueLength metris will increase to 1k+  because of slow consumption with lock competition.  See details code ： BPServiceActor.CommandProcessingThread

And Namenode  distribute all replicationWork to datanode through heartbeat.  All LowerReconstruction blocks will in neededReconstruction.  Every heartbeat will distribute liveDatnaodes*DFS_NAMENODE_REPLICATION_WORK_MULTIPLIER_PER_ITERATION to the related datanodes.  But as mentioned above，the datanode blocked executing the transfer block commands.  When 5min later by default ,  the pendingReconstruction will into timeout and then re-entry neededReconstruction, and then re-entry pendingReconstruction . So the datanodes will receive multiple transfer command with same block.  That will lead to excess blocks.

Continuous cycle，it will lead to serious consequences.  Datanode has more tranfers commands in queue and waste more disk space  and result in more excess blocks.

So, I think we should limit the max size of pendingReconstruction.