Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47199

Add prefix into TemporaryDirectory to avoid flakiness

    XMLWordPrintableJSON

Details

    Description

      Sometimes the test fail because the temporary directory names are same (https://github.com/apache/spark/actions/runs/8066850485/job/22036007390).

      File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line ?, in pyspark.sql.dataframe.DataFrame.writeStream
      Failed example:
          with tempfile.TemporaryDirectory() as d:
              # Create a table with Rate source.
              df.writeStream.toTable(
                  "my_table", checkpointLocation=d)
      Exception raised:
          Traceback (most recent call last):
            File "/usr/lib/python3.11/doctest.py", line 1353, in __run
              exec(compile(example.source, filename, "single",
            File "<doctest pyspark.sql.dataframe.DataFrame.writeStream[3]>", line 1, in <module>
              with tempfile.TemporaryDirectory() as d:
            File "/usr/lib/python3.11/tempfile.py", line 1043, in __exit__
              self.cleanup()
            File "/usr/lib/python3.11/tempfile.py", line 1047, in cleanup
              self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
            File "/usr/lib/python3.11/tempfile.py", line 1029, in _rmtree
              _rmtree(name, onerror=onerror)
            File "/usr/lib/python3.11/shutil.py", line 738, in rmtree
              onerror(os.rmdir, path, sys.exc_info())
            File "/usr/lib/python3.11/shutil.py", line 736, in rmtree
              os.rmdir(path, dir_fd=dir_fd)
          OSError: [Errno 39] Directory not empty: '/__w/spark/spark/python/target/4f062b09-213f-4ac2-a10a-2d704990141b/tmp29irqweq'
      

      Attachments

        Issue Links

          Activity

            People

              gurwls223 Hyukjin Kwon
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: