RpcConnection could still remain opened after we shutdown the NettyRpcServer

The log is very strange, we keep sending request to a dead RS, and the result is not connection refused, but rpc timeout, and later it becomes CallQueueTooBig...