Improve HBaseFsck Scalability

There are some problems with HBaseFsck that make it unnecessarily slow especially for large tables or clusters with many regions.

This patch tries to fix the biggest bottlenecks and also include a couple of bug fixes for some of the race conditions caused by gathering and holding state about a live cluster that is no longer true by the time you use that state in Fsck processing. These race conditions cause Fsck to crash and become unusable on large clusters with lots of region splits/merges.

Here are some scalability/performance problems in HBaseFsck and the changes the patch makes:

• Unnecessary I/O and RPCs caused by fetching an array of FileStatuses and then discarding everything but the Paths, then passing the Paths to a PathFilter, and then having the filter look up the (previously discarded) FileStatuses of the paths again. This is actually worse than double I/O because the first lookup obtains a batch of FileStatuses while all the other lookups are individual RPCs performed sequentially.
  • Avoid this by adding a FileStatusFilter so that filtering can happen directly on FileStatuses
  • This performance bug affects more than Fsck, but also to some extent things like snapshots, hfile archival, etc. I didn't have time to look too deep into other things affected and didn't want to increase the scope of this ticket so I focus mostly on Fsck and make only a few improvements to other codepaths. The changes in this patch though should make it fairly easy to fix other code paths in later jiras if we feel there are some other features strongly impacted by this problem.
• OfflineReferenceFileRepair is the most expensive part of Fsck (often 50% of Fsck runtime) and the running time scales with the number of store files, yet the function is completely serial
  • Make offlineReferenceFileRepair multithreaded
• LoadHdfsRegionDirs() uses table-level concurrency, which is a big bottleneck if you have 1 large cluster with 1 very large table that has nearly all the regions
  • Change loadHdfsRegionDirs() to region-level parallelism instead of table-level parallelism for operations.

The changes benefit all clusters but are especially noticeable for large clusters with a few very large tables. On our version of 0.98 with the original patch we had a moderately sized production cluster with 2 (user) tables and ~160k regions where HBaseFsck went from taking 18 min to 5 minutes.