Detecting multiple test-frameworks

When multiple test frameworks are present on project classpath surefire chooses one framework for running test.

During automatically provider discover some of type test can be skipped without any knowledge and information for developer.

It is danger situation than developer can add second test framework to project and in the result some of test will be skipped silently.

We can force usage one or more needed providers by adding those to plugin dependency and we can have control on executing test.

Proposition

• add options multipleFrameworks empty (not set) by default, with possible value warn or fail

Scenario 1 - default

when

• multipleFrameworks = empty - not set
• many tests framework on classpath
• no providers on plugin dependency

then

• current behavior - choose first provider without warning about skipped test

Scenario 2

when

• multipleFrameworks = warn
• many tests framework on classpath
• no providers on plugin dependency

then

• print warn about multiple providers

Scenario 3

when

• multipleFrameworks = fail
• many tests framework on classpath
• no providers on plugin dependency

then

• print error about multiple providers and break build

Scenario 4

when

• many tests framework on classpath
• selected providers on plugin dependency

then

• ignore multipleFrameworks
• use providers from dependency list, no warning