Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-12735

Execute flow analysis before committing flow into Registry

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-M3
    • Core Framework
    • None

    Description

      Based on https://lists.apache.org/thread/hsg2ggj9p47lvj5m00okkj0ok58zdoqp

      When committing a flow into a Registry, it would be beneficial to have the opportunity to execute the existing flow analysis rules and potentially interrupt the commit.

      The change aims for:

      • Execute existing flow analysis rules out of the scheduled times
      • Gracefully resject commit when the flow does not meet the requirements
      • Providing a toggle to turn on or off the feature
      • In case of the feature is inactive or no rules are set, the Registry handling behaviour should not change*

      The change does not aim for:

      • RegistryClient level adjustments
      • Using different rules set or other kind of validation methods
      • Providing support for the 1.x line

      {*}
      Expect this following branch: https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5028
      As with the usage of this feature, failing snapshot commits will be more frequent and I consider "lingering" flow definitions without snapshot misleading for users.

      The following test cases describe the proposed behaviour in detail:

      TC1.1 Adding flow with rules violation
      GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND NiFi property `nifi.registry.check.for.rules.violation` is true
      AND process group G1 containing a `GenerateFlowFile`
      AND violation appears as validation failure
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the UI displays the following error message "Cannot store flow version to registry due to rules violations"
      AND the flow will not appear in registry R1
      AND the UI will show G1 as un-versioned
      AND NiFi node will see G1 as un-versioned

      TC1.2 Adding flow with rules violation before scheduled
      GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND NiFi property `nifi.registry.check.for.rules.violation` is true
      AND process group G1 containing a `GenerateFlowFile`
      AND violation does not appear in UI
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the UI displays the following error message "Cannot store flow version to registry due to rules violations"
      AND the flow will not appear in registry R1
      AND the UI will show G1 as un-versioned
      AND NiFi node will see G1 as un-versioned

      TC1.3 When turned off violating flows will be added
      GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND NiFi property `nifi.registry.check.for.rules.violation` is fals
      AND process group G1 containing a `GenerateFlowFile`
      AND violation appears as validation failure
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the flow is committed normally

      TC1.4 Adding version with rules violation (when the rule is new)
      GIVEN NiFi property `nifi.registry.check.for.rules.violation` is true
      AND process group G1 containing a `GenerateFlowFile`
      AND process group G1 is under version control
      AND `LogAttribute` processor is added (not committed to the registry)
      GIVEN adding a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND the user tries to add G1 under version control, using registry R1
      THEN the UI displays the following error message "Cannot store flow version to registry due to rules violations"
      AND the flow will not appear in registry R1
      AND the UI will show G1 as un-versioned
      AND NiFi node will see G1 as un-versioned

      TC1.5 Adding version with rules violation (when the violation is new)
      GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND NiFi property `nifi.registry.check.for.rules.violation` is true
      AND process group G1 is under version control
      AND `GenerateFlowFile` processor is added (not committed to the registry)
      WHEN the user tries to add new version of G1 to registry R1
      THEN the UI displays the following error message "Cannot store flow version to registry due to rules violations"
      AND the flow will not appear in registry R1
      AND the UI will show G1 as un-versioned
      AND NiFi node will see G1 as un-versioned

      TC2.1 Adding flow after fixing violation (by removing violating processor)
      CONTINUING TC1.1
      WHEN `GenerateFlowFile` is deleted from process group G1
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the flow is committed normally

      TC2.2 Adding flow after fixing violation (by disabling violated rule)
      CONTINUING TC1.5
      WHEN `DisallowComponentType` is disabled
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the flow is committed normally

      TC2.3 Adding version after fixing violation (by removing violating processor)
      CONTINUING TC1.5
      WHEN `GenerateFlowFile` is deleted from process group G1
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the flow is committed normally

      TC2.4 Adding version after fixing violation (by disabling violated rule)
      CONTINUING TC1.5
      WHEN `DisallowComponentType` is disabled
      WHEN the user tries to add G1 under version control, using registry R1
      THEN the flow is committed normally

      TC3.1 Adding violating flow after disabling rule
      GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND NiFi property `nifi.registry.check.for.rules.violation` is true
      AND process group G1 containing a `GenerateFlowFile`
      AND violation appears as validation failure
      WHEN `DisallowComponentType` is disabled
      AND the user tries to add G1 under version control, using registry R1
      THEN the flow is committed normally

      TC3.2 Adding violating version after turning disabling rule
      GIVEN a `DisallowComponentType` flow analysis rule, disallowing `GenerateFlowFile`
      AND NiFi property `nifi.registry.check.for.rules.violation` is true
      AND process group G1 is under version control
      AND `GenerateFlowFile` processor is added (not committed to the registry)
      AND violation appears as validation failure
      WHEN `DisallowComponentType` is disabled
      AND the user tries to add new version of G1 to registry R1
      THEN the flow is committed normally

      Attachments

        Issue Links

          Activity

            People

              simonbence Simon Bence
              simonbence Simon Bence
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m