Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6148

RunMongoAggregation - Add support for allowDiskUse option

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.9.0
    • 1.10.0
    • Core Framework
    • All
    • Patch

    Description

      Currently the RunMongoAggregation processor does not allow the use of the 'allowDiskUse' aggregation option. This is required when aggregation pipeline stages exceed the maximum memory use limit (100MB) and is typically exceeded when working with large datasets particularly when executing $group or $sort operations.

       

      Pull Request will submitted in due course to add a mandatory property on the RunMongoAggregation processor (defaulting to false).

       

      https://docs.mongodb.com/manual/reference/command/aggregate/#aggregate-data-using-external-sort

      http://mongodb.github.io/mongo-java-driver/3.10/javadoc/com/mongodb/client/AggregateIterable.html#allowDiskUse-java.lang.Boolean-

       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            cdavid15 Craig Davidson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 0.5h
              0.5h

              Slack

                Issue deployment