Instrumentation for read/write locks in LLAP

LLAP has several R/W locks for serialization of updates to query tracker, file data, ....

Instrumentation is added to monitor the

• total amount of R/W locks within a particular category
• average + max wait/suspension time to get the R/W lock

A category includes all lock instances for particular areas (i.e. category is FileData and all R/W locks that are used in FileData instances are accounted within the one category).

The monitoring/accounting is done via Hadoop Metrics 2, making them accessible via JMX. In addition, a new "locking" GET endpoint is added to the LLAP daemon's REST interface. It produces output like the following example:

{
  "statsCollection": "enabled",
  "lockStats": [
    { "type": "R/W Lock Stats",
      "label": "FileData",
      "totalLockWaitTimeMillis": 0,
      "readLock": {
         "count": 0,
         "avgWaitTimeNanos": 0,
         "maxWaitTimeNanos": 0
      },
      "writeLock": {
         "count": 0,
         "avgWaitTimeNanos": 0,
         "maxWaitTimeNanos": 0
              }
    },
    { "type": "R/W Lock Stats",
      "label": "QueryTracker",
      "totalLockWaitTimeMillis": 0,
      "readLock": {
         "count": 0,
         "avgWaitTimeNanos": 0,
         "maxWaitTimeNanos": 0
      },
      "writeLock": {
         "count": 0,
         "avgWaitTimeNanos": 0,
         "maxWaitTimeNanos": 0
              }
    } ]
}

To avoid the overhead of lock instrumentation, lock metrics collection is disabled by default and can be enabled via the following configuration parameter:
 hive.llap.lockmetrics.collect = true