Unit tests silently failing

I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.

I'm working on a NetBeans platform application and we recently noticed that our unit tests are silently failing. Unit tests fail but `ant test` completes with exit code 0. We are running this through Travis which marks the PR as if it passed.

Based on debugging the issue broadly is that the continue.after.failing.tests property (as defined in ${NETBEANS}/harness/README) does not work.
 
The reason it doesn't work is that the ant target used for (-do-testng in ${NETBEANS}/harness/common.xml) isn't processing it correctly
 

773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail> 

 
The unless keyword will pass if the parameter exists (even if it has no value or a false value), so it doesn't matter if you set it true or false, the fail state will never happen.
 
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always exists even if you don't set it.
 

584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback --> 

 
You can re-produce this issue simply by creating a new module suite containing one module and one unit test which is set to always fail.
 
 

import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

public class AntTestNGTest {
    
    /**
     * Test of main method, of class AntTest.
     */
    @Test
    public void testMain() throws InterruptedException {
        System.out.println("main");
        String[] args = null;
        AntTest.main(args);
        // TODO review the generated test code and remove the default call to fail.
        fail("The test case is a prototype.");
    }
}
 

 
Then run:
 

ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false test

​
or

ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false -f AntTestModule test-unit

or add continue.after.failing.tests=false to the {{project.properties }}file for your module suite and test through Netbeans.
​
Either way, the test will fail but the build will succeed.
​
If you then change the unless condition in ${NETBEANS}/harness/common.xml to point to a property which does not exist
 

773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some tests failed; see details above.</fail> 

 
The build will fail with the message "Some tests failed..." as expected.
 A suggested fix would be to add some logic prior to this fail condition to check "continue.after.failing.tests" and create a new property if it is set to true - that way you could check for that property in the unless condition and it would either exist (fail does not happen, all tests are executed) or not-exist (fail state will happen if tests failed).
​
Also, while we're at it - I also confirmed the test.timeout property (also defined in ${NETBEANS}/harness/README) also does not work - probably for similar reasons.