Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8002

Create new Elasticsearch Scroll/Paginated Query processor using REST API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.12.1
    • 1.15.0
    • None
    • None

    Description

      The existing ScrollElasticsearch processor does not fully work with newer versions of Elasticsearch (e.g. 7.x).

      A new version of this processor should be created using the REST API approach, allowing for a query to be passed as input FlowFile content. Appropriate parameters should be set as processor properties (e.g. index, scroll size, scroll time) and allow for additional query string parameters to be set as dynamic properties (see NIFI-8001).

      The processor should support both _scroll and _pit/search_after. While scroll may be backward compatible, _pit was introduced in ES 7 (an earlier version of search_before exists for <7, but suggest not trying to support that as part of this ticket).

      It may be possible to simply extend the existing REST API's Query processor to page through results with the selected method, rather than create a new processor.

      Attachments

        1. ES_Paginated_Queries.json
          55 kB
          Chris Sampson
        2. ES_Paginated_Queries.xml
          124 kB
          Chris Sampson

        Issue Links

          Activity

            People

              Chris S Chris Sampson
              Chris S Chris Sampson
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 5h Original Estimate - 5h
                  5h
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 57.5h
                  57.5h