Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30143

StreamingQuery.stop() should not block indefinitely

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Done
    • 2.4.4
    • 3.0.0
    • Structured Streaming
    • None

    Description

      The stop() method on a Streaming Query awaits the termination of the stream execution thread. However, the stream execution thread may block forever depending on the streaming source implementation (like in Kafka, which runs UninterruptibleThreads).

      This causes control flow applications to hang indefinitely as well. We'd like to introduce a timeout to stop the execution thread, so that the control flow thread can decide to do an action if a timeout is hit. 

      Attachments

        Issue Links

          Activity

            People

              brkyvz Burak Yavuz
              brkyvz Burak Yavuz
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: