Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3696

MQTT IO should compute watermark and ack messages outside of finalizeCheckpoint method

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • 2.2.0
    • None
    • io-java-mqtt
    • None
    • - Flink - beam-runners-flink_2.10:2.2.0
      - Beam and related jars - 2.2.0

    Description

      I'm experiencing a situation when an incoming message isn't acknowledged (therefore in sometime broker resend it) and the watermark is not updated while new messages are coming continuously.

      After some time I've discovered that this situation is related to the fact that finalizaCheckpoint is not being called.

      I took a look at the Pubsub IO implementation and found that they expect such situation and do not compute watermark and ack messages in finalizeCheckpoint. Here is the comment about that: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L289

      Should MQTT IO do the same?

      Attachments

        Activity

          People

            Unassigned Unassigned
            kolchinmax Maxim Kolchin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: