For ServerCrashProcedure, TransitRegionStateProcedure and related region assignment procedures like OpenRegionProcedure/CloseRegionProcedure, now we will try our best to release the procedure worker when doing time consuming operations. Including:
1. When updating region state to hbase:meta. We will use async client to get a CompletableFuture, and then suspend the procedure. When the CompletableFuture is done, we add the procedure back.
2. When acquiring region state node lock, if we can not get the lock immediately, we will also suspend the procedure and release the procedure worker. Once it is our turn to get the lock, we will add the procedure back.
In this way, if meta is offline, it is not likely that the assign procedures will consume all the procedure workers and cause dead lock. You can see
HBASE-19976 for more detailed description about this problem.