Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1878

Add failOnFlakeCount option

    XMLWordPrintableJSON

Details

    Description

      The Surefire plugin currently covers two use cases with its `rerunFailingTestCount` option:

      1. rerunFailingTestCount=0: The build will fail immediately, but I don't know if it was a real failure or a flake. This option is best for healthy code bases with very little or no flakiness.
      2. rerunFailingTestCount>0: I will know when a test is flaky, but the build will be successful, so there is no pressure to fix the flakes. This helps when the situation is really bad and I just want to get some green build again to boost team morale

      I'd like to support a third use case:

      3. rerunFailingTestCount>0 and a new option failOnFlakeCount > 0: The build will fail (so there is actually pressure to deal with flakiness) and I can tell the difference between real failures and flakes. I think this would be the best option for projects with a medium amount of flaky tests and a team that wants to take care of them.

      Edit: To clarify, this is about failing the whole test set when there is flakiness, not individual tests.

      Attachments

        Activity

          People

            tibordigana Tibor Digana
            oehme Stefan Oehme
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: