Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27499

Add optional table name filter to the HMS notification fetch API

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0-beta-1
    • None

    Description

      The current notification fetch API does not support filters on table name:

      struct NotificationEventRequest {
          1: required i64 lastEvent,
          2: optional i32 maxEvents,
          3: optional list<string> eventTypeSkipList,
      }
      NotificationEventResponse get_next_notification(1:NotificationEventRequest rqst)
      

      Engines that cache HMS metadata and file metadata might want to know whether they have synced to the latest events on some given tables.

      In order to efficiently fetch events specific to a table, we should enhance the notification fetch API to include an optional table name in the request so that only events for that given table are returned. The NOTIFICATION_LOG table in the HMS should also have an index on table_name so that API calls with the filter do not do a table scan.

      With the index added, we can also extend the NotificationEventsCountRequest with an optional table name, and extend the response to carry the largest event id on that table. Currently they are:

      struct NotificationEventsCountRequest {
          1: required i64 fromEventId,
          2: required string dbName,
          3: optional string catName,
          4: optional i64 toEventId,
          5: optional i64 limit
      }
      struct NotificationEventsCountResponse {
          1: required i64 eventsCount,
      }
      NotificationEventsCountResponse get_notification_events_count(1:NotificationEventsCountRequest rqst)
      

      Similar API for multiple tables (given a table lists) will also be helpful since a query usually involves several tables.

      CC hemanth619, ngangam 

      Attachments

        Issue Links

          Activity

            People

              hemanth619 Sai Hemanth Gantasala
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: