Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12959

Support loading Python processors from NARs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-M3
    • Core Framework
    • None

    Description

      Currently, third-party dependencies for Python Processors can be handled in two ways. Either they can be declared as dependencies in a Processor itself; or the Processor can be in a module where a requirements.txt dictates the requirements. These can be very helpful for developing Python based Processors.

      However, in production environments, it is not uncommon to see environments where pip is not installed. There is an inherent risk in allowing remote code to be downloaded in an ad-hoc manner like this, without any sort of vulnerability scanning, etc.

      As such, we should allow users to also package python packages in NiFi's native archiving format (NARs).

      The package structure should be as follows:

      my-nar.nar
      +-- META-INF/
          +-- MANIFEST.MF
      +-- NAR-INF/
          +-- bundled-dependencies/
              +-- dependency1
              +-- dependency2
              +-- etc.
      +-- MyProcessor.py

      Where MyProcessor.py could also be a python module / directory.

      In this way, we allow a Python Processor to be packaged up with its third party dependencies and dropped in the lib/ directory (or extensions) directory of a NiFi installation in the same way that a Java processor would be.

      Attachments

        Issue Links

          Activity

            People

              markap14 Mark Payne
              markap14 Mark Payne
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m