Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.0.0-M8
-
None
-
None
Description
Summary
If a JUnit5 test class fails during setup and succeeds on retry, then the surefire test goal and entire build will fail. On the other hand, if the failure occurs in a test method which succeeds on retry, then the goal and the build will succeed.
Reproducer
Test class with flaky setup:
package example; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; import java.io.File; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; public class FlakyClassTest { @BeforeAll public static void setup() throws IOException { String testSetupMarkerFile = "testSetupMarker.txt"; if (!new File(testSetupMarkerFile).exists()) { System.out.println("I'm failing!"); Files.write(Paths.get(testSetupMarkerFile), "Hello".getBytes()); throw new RuntimeException("I'm failing!"); } else { System.out.println("I'm passing!"); } } @Test public void test() { } }
Test class with flaky method:
package example; import org.junit.jupiter.api.Test; import java.io.File; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; public class FlakyMethodTest { @Test public void test() throws IOException { String testMethodMarkerFile = "testMethodMarker.txt"; if (!new File(testMethodMarkerFile).exists()) { System.out.println("I'm failing!"); Files.write(Paths.get(testMethodMarkerFile), "Hello".getBytes()); throw new RuntimeException("I'm failing!"); } else { System.out.println("I'm passing!"); } } }
Surefire config:
<plugin> <artifactId>maven-surefire-plugin</artifactId> <version>3.0.0-M8</version> <configuration> <rerunFailingTestsCount>1</rerunFailingTestsCount> </configuration> </plugin>
Actual behavior
1. This execution succeds:
mvn test -Dtest=*FlakyMethod*
2. This execution fails:
mvn test -Dtest=*FlakyClass*
Expected behavior
1. Both executions succeed