Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      As pointed out by Jose Torres in https://github.com/apache/spark/pull/20936, both epoch queue and commits/offsets maps are unbounded by the number of waiting epochs. According to his proposal, we should introduce some configuration option for maximum epoch backlog and report an error if the number of waiting epochs exceeds it.

        Attachments

          Activity

            People

            • Assignee:
              gsomogyi Gabor Somogyi
              Reporter:
              efim.poberezkin Efim Poberezkin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: