Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12693

When Processor is removed, Python Process should be notified asynchronously

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-M3
    • Core Framework
    • None

    Description

      When a Processor is removed, the PythonBridge is notified of the removal, and it then notifies any relevant Python process. This is done synchronously during the removal. I encountered two occurrences in which notifying the Python process failed.

      While the failure itself is not a huge concern, the handling of those failures resulted in very bad outcomes. In the first instance, the communication with the Python process was blocked on a socket read or write. As a result, the Service Facade's lock was never released, and no web requests could be made; they all blocked on the read lock. This resulted in requiring a restart of NiFi.

      In the other scenario, the call did not block indefinitely but threw an Exception. In this case, the associated Connections were never removed. As a result, I could no longer navigate to that Process Group in the UI, or the UI would have errors because there were Connections whose source or destination didn't exist. This required manually removing those connections from the flow.json file to recover.

      Since the intention of this action is simply a notification so that the Python process can cleanup after itself, this notification should be moved to a background thread, so that any failures are simply logged without causing problematic side effects.

      Attachments

        Issue Links

          Activity

            People

              markap14 Mark Payne
              markap14 Mark Payne
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m