Create table is slow in large cluster when AccessController is enabled.

Problem:

In HBase 1.3.x  large, performance test, cluster (100 RS, 60k tables, 600k regions) a simple table creation takes around 150 seconds. The time taken varies but still takes lot of time.

Analysis:

1. When HBase creates a table , it calls AssignmentManager#assign(final ServerName destination, final List<HRegionInfo> regions)
In AssignmentManager#assign，it calls asyncSetOfflineInZooKeeper(state, cb, destination), and waits in below code loop for 2 minutes. 

 if (useZKForAssignment) {
          // Wait until all unassigned nodes have been put up and watchers set.
          int total = states.size();
          for (int oldCounter = 0; !server.isStopped();) {
            int count = counter.get();
            if (oldCounter != count) {
              LOG.debug(destination.toString() + " unassigned znodes=" + count +
                " of total=" + total + "; oldCounter=" + oldCounter);
              oldCounter = count;
            }
            if (count >= total) break;
            Thread.sleep(5);
          }
        }

2. asyncSetOfflineInZooKeeper creates a znode under /hbase/region-in-transition/ and calls exist to ensure that znode is created. This is simple operation should not take much time. Then where the time it taken!!!

3. ZooKeeper client API process watcher notification and async API response through a queue one by one.
If there is a delay in any watcher/response processing by the client, in this case HBase, all other response processing is delayed. Then it appears as if API call has taken more time.
Same thing happen in this issue.

Watcher processing for znode creation under /hbase/acl took most of the time and delayed /hbase/region-in-transition/region znode creation processing. This is why wait in loop was too long. 

4. Watcher processing for znode creation under hbase/acl/ calls ZKPermissionWatcher#nodeChildrenChanged, which internally calls ZKUtil.getChildDataAndWatchForNewChildren
which calls ZooKeeper's getData API, in this use case, 60k times which takes most of the time.

Solutions:
Move getChildDataAndWatchForNewChildren call into the async code block in ZKPermissionWatcher#nodeChildrenChanged.