Details
Major Description
IndexShuffleBlockResolverSuite fails in my windows develop machine.
[info] IndexShuffleBlockResolverSuite: [info] - commit shuffle files multiple times *** FAILED *** (388 milliseconds) [info] Array(10, 0, 20) equaled Array(10, 0, 20) (IndexShuffleBlockResolverSuite.scala:108) [info] org.scalatest.exceptions.TestFailedException: ..... ..... [info] Exception encountered when attempting to run a suite with class name: org.apache.spark.shuffle.sort.IndexShuffleB lockResolverSuite *** ABORTED *** (2 seconds, 234 milliseconds) [info] java.io.IOException: Failed to delete: C:\Users\yyu29\Documents\codes.next\spark\target\tmp\spark-0e81a15a-e712 -4b1c-a089-f421db149e65 [info] at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:940) [info] at org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala: 60) [info] at org.scalatest.BeforeAndAfterEach$class.afterEach(BeforeAndAfterEach.scala:205) [info] at org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala: 36) [info] at org.scalatest.BeforeAndAfterEach$class.afterEach(BeforeAndAfterEach.scala:220) [info] at org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala: 36)
Root cause is when "afterEach" wants to clean up data, some files are still open. For example:
// The dataFile should be the previous one val in = new FileInputStream(dataFile) val firstByte = new Array[Byte](1) in.read(firstByte) assert(firstByte(0) === 0)
Lack of "in.close()".
In Linux, it is not a problem, you can still delete a file even it is open, but this does not work in windows, which will report "resource is busy".
Another issue is this IndexShuffleBlockResolverSuite.scala is a scala file but it is placed in "test/java".
Attachments
Issue Links
- links to
