Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2185

KafkaIO bounded source

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • io-java-kafka
    • None

    Description

      KafkaIO could be a useful source for batch applications as well. It could implement a bounded source. The primary question is how the bounds are specified.

      One option : Source specifies a time period (say 9am-10am), and KafkaIO fetches appropriate start and end offsets based on time-index in Kafka. This would suite many batch applications that are launched on a scheduled.

      Another option is to always read till the end and commit the offsets to Kafka. Handling failures and multiple runs of a task might be complicated.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-6466 KafkaIO doesn't commit offsets while being used as bounded source

          • P1 - Critical bug: data loss, total loss of function, or loss of testing signal due to test failures or flakiness
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. BEAM-2248 KafkaIO support to use start read time to set start offset

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              rangadi Raghu Angadi
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: