XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      Currently, the EpochCoordinator doesn't enforce a commit order. If a message for epoch n gets lost in the ether, and epoch n + 1 happens to be ready for commit earlier, epoch n + 1 will be committed.

       

      This is either incorrect or needlessly confusing, because it's not safe to start from the end offset of epoch n + 1 until epoch n is committed. EpochCoordinator should enforce this sequencing.

       

      Note that this is not actually a problem right now, because the commit messages go through the same RPC channel from the same place. But we shouldn't implicitly bake this assumption in.

        Attachments

          Activity

            People

            • Assignee:
              efim.poberezkin Efim Poberezkin
              Reporter:
              joseph.torres Jose Torres
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: