Hierarchical metastore event processing

Current Issue:

At present, metastore event processor is single threaded. Notification events are processed sequentially with a maximum limit of 1000 events fetched and processed in a single batch. Multiple locks are used to address the concurrency issues that may arise when catalog DDL operation processing and metastore event processing tries to access/update the catalog objects concurrently. Waiting for a lock or file metadata loading of a table can slow the event processing and can affect the processing of other events following it. Those events may not be dependent on the previous event. Altogether it takes a very long time to synchronize all the HMS events.

Proposal:

Existing metastore event processing can be turned into multi-level event processing. Idea is to segregate the events based on their dependency, maintain the order of events as they occur within the dependency and process them independently as much as possible:

1. All the events of a table are processed in the same order they have actually occurred.
2. Events of different tables are processed in parallel.
3. When a database is altered, all the events relating to the database(i.e., for all its tables) occurring after the alter db event are processed only after the alter database event is processed ensuring the order.

Have attached an initial proposal design document

https://docs.google.com/document/d/1KZ-ANko-qn5CYmY13m4OVJXAYjLaS1VP-c64Pumipq8/edit?pli=1#heading=h.qyk8qz8ez37b