Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
ghx-label-11
Description
The inflight event list for catalog versions could have dangling items, e.g. in some catalog operations we don't trigger any HMS APIs but still add the version into the list.
To be more robust, we should add GC to InFlightEvents:
- For idsForInflightDmlEvents_, if the event id is smaller than the lastSycnedEventId of the event-processor, it won't be matched anymore.
- For versionsForInflightEvents_, it's a bit complex. We don't know when the catalog version is stale. Maybe we should refactor the self-event detection to only check the catalog service id and ignore any event that have older catalog version than the current table. The possibility of two Impala clusters having the same catalog service id is pretty small.