Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47734

Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query

    XMLWordPrintableJSON

Details

    Description

      https://issues.apache.org/jira/browse/SPARK-47199 didn't fix the flakiness in the pyspark.sql.dataframe.DataFrame.writeStream doctest : the problem is not that we are colliding on the test but, rather, that the test is starting a background thread to write to a directory then deleting that directory from the main test thread, something which is inherently race prone.

      The fix is simple: stop the streaming query in the doctest itself, similar to other streaming doctest examples.

      Attachments

        Issue Links

          Activity

            People

              joshrosen Josh Rosen
              joshrosen Josh Rosen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: