Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3730

Allow exclusion of begin/commit events in CaptureChangeMySQL

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.2.0
    • Extensions
    • None

    Description

      Notes from ijokarumawak:

      In an end-to-end CDC flow, we would use CaptureChangeMySQL to get database events to stream into PutSQL or PutDatabaseRecord. Those database events include SQL statements such as begin and commit, and such statements are executed literally here.

      I think it would be great if users can configure CaptureChangeMySQL whether to emit begin and commit event or not, as these statements do not have significant meaning for synchronizing change since CaptureChangeMySQL emits FlowFile per updated record, but just for EnforceOrder to order events correctly.

      If we can eliminate these events, we can minimize the number of FlowFiles which would lead us to a better performance.

      Also, (for MySQL at least) begin and commit is not a database or table specific event. When I replicate changes from table A to table B using CaptureChangeMySQL and PutDatabaseRecord, I saw following behavior and felt begin and commit are a little bit disturbing:

      Insert a row into table A
      3 events are emitted, begin, insert A and commit. Then CaptureChangeMySQL emitted 3 FlowFiles.
      PutDatabaseRecord execute 3 SQLs. begin, insert B and commit.
      Another 3 events are emitted derived from insert B via MySQL bin-log, begin, insert B and commit.
      Since I have configured CaptureChangeMySQL only listens to table A, it only emitted 2 FlowFiles, begin and commit.
      PutDatabaseRecord executes the begin and commit SQL.

      Attachments

        Issue Links

          Activity

            People

              mattyb149 Matt Burgess
              mattyb149 Matt Burgess
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: