Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7473

Add @SupportsBatching to Azure storage processors

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Extensions

    Description

      Some stuff to think through would be if there was one failure in a batch of many and it all ran again is the result the same and does the SDK throw any exceptions?

      For a process that's just modifying some content or adding attributes it's as simple as adding the annotation.
      But when it modifies some external resource you sometimes have to take care to make sure it'd do the right thing if there's a failure.
      I think the Fetch won't require any code changes. Not sure about Delete or Put.

       

      https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html

      SupportsBatching: This annotation indicates that it is okay for the framework to batch together multiple ProcessSession commits into a single commit. If this annotation is present, the user will be able to choose whether they prefer high throughput or lower latency in the Processor’s Scheduling tab. This annotation should be applied to most Processors, but it comes with a caveat: if the Processor calls ProcessSession.commit, there is no guarantee that the data has been safely stored in NiFi’s Content, FlowFile, and Provenance Repositories. As a result, it is not appropriate for those Processors that receive data from an external source, commit the session, and then delete the remote data or confirm a transaction with a remote resource.

       

       

       

      Attachments

        Issue Links

          Activity

            People

              sjyang18 Seokwon Yang
              sjyang18 Seokwon Yang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m