Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6168

Allow modification of JSON value before writing to ElasticSearch

Details

    • Improvement
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-elasticsearch
    • None

    Description

      I have an Apache Beam streaming job which reads data from Kafka and writes to ElasticSearch using ElasticSearchIO.

      The issue I'm having is that messages in Kafka already have key field, and using ElasticSearchIO.Write.withIdFn() I'm mapping this field to document _id field in ElasticSearch.

      Having a big volume of data I don't want the key field to be also written to ElasticSearch as part of _source.

      Is there an option/workaround that would allow doing that?

      Attachments

        Activity

          People

            Unassigned Unassigned
            marknorkin Mark Norkin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: