Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12601

Support append-only indices in ES output

Details

    • Improvement
    • Status: Triage Needed
    • P2
    • Resolution: Done
    • None
    • 2.33.0
    • io-java-elasticsearch
    • None

    Description

      Currently, the Apache Beam Elasticsearch sink is using the index bulk API operation to add data to the target index. When using append-only indices it is better to use the create operation. This also applies to new append-only indexes, like data streams.

      The scope of this improvement is to add a new boolean configuration option, append-only, to the Elasticsearch sink, with a default value of false (to keep the current behaivour) that when enabled makes it use the create operation instead of the index one when sending data.

      Attachments

        Activity

          People

            andresrc Andres Rodriguez
            andresrc Andres Rodriguez
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 10m
                5h 10m