Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.2.0
    • None
    • None
    • ghx-label-1

    Description

      Here is a the logic to filter out create events from a given batch of events.

      Iterator<MetastoreEvent> it = metastoreEvents.iterator();
            // filter out the create events which has a corresponding drop event later
            int fromIndex = 0;
            int numFilteredEvents = 0;
            int inputSize = metastoreEvents.size();
            while (it.hasNext()) {
              MetastoreEvent current = it.next();
              if (fromIndex < metastoreEvents.size() && current.isRemovedAfter(
                  metastoreEvents.subList(fromIndex + 1, metastoreEvents.size()))) {
                LOG.info(current.debugString("Filtering out this event since the object is "
                    + "either removed or renamed later in the event stream"));
                it.remove();
                numFilteredEvents++;
              }
              fromIndex++;
            }
      

      If the event list contains CREATE_DATABASE, CREATE_TABLE, DROP_TABLE, DROP_DATABASE events, it is possible that we only filter out CREATE_DATABASE event and not CREATE_TABLE event. This is because the fromIndex above gets incremented event iteration of the while loop and hence when it is evaluating if there is a inverse event later on in the stream, it starts to look from DROP_DATABASE event onwards.

      The fix is simple, the fromIndex needs to be incremented only when the item from the list is not removed.

      Attachments

        Issue Links

          Activity

            People

              vihangk1 Vihang Karajgaonkar
              vihangk1 Vihang Karajgaonkar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: