Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1914

XML report omits method signature / display name of Junit 5 parameterized tests if testset reporter is configured to use phrased naming

    XMLWordPrintableJSON

Details

    Description

      Description of the issue

      Given a test class with two parameterized tests as follows:

          @ParameterizedTest
          @ValueSource(strings = {"a", "b"})
          void test1(String param) {
              // test code
          }
      
          @ParameterizedTest
          @ValueSource(strings = {"a", "b"})
          void test2(String param) {
              // test code
          }
      

      and a surefire configuration that includes the following:

      <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
          <version>3.0</version>
          <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
          <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
          <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
      </statelessTestsetReporter>
      

      the XML report will look like this:

        <testcase name="[1] a" classname="mypackage.MyClass" time="0.001" />
        <testcase name="[2] b" classname="mypackage.MyClass" time="0.001" />
        <testcase name="[1] a" classname="mypackage.MyClass" time="0.001" />
        <testcase name="[2] b" classname="mypackage.MyClass" time="0.001" />
      

      Note: The test method name/signature is not included in the name attribute.

      I would expect something more like this:

        <testcase name="test1(String) [1] a" classname="mypackage.MyClass" time="0.001" />
        <testcase name="test1(String) [2] b" classname="mypackage.MyClass" time="0.001" />
        <testcase name="test2(String) [1] a" classname="mypackage.MyClass" time="0.001" />
        <testcase name="test2(String) [2] b" classname="mypackage.MyClass" time="0.001" />
      

      Some context on why this is bad

      Omitting the test method name makes it impossible to differentiate individual test cases in the XML report if there are multiple @ParameterizedTest-annotated test cases in the same class that use the same parameters.

      Any software that parses the surefire XML reports will be misled into thinking these are multiple executions of the same test.

      Variant with @DisplayName usage

      There is a variant of this issue: If the @ParameterizedTest has a @DisplayName annotation, whatever was chosen as a display name is not included in the XML report either - unless it is explicitly referenced in the name attribute of the @ParameterizedTest like so: @ParameterizedTest(name = "{displayName} ...").

      Solution ideas

      This issue has already been brought up during the discussion of SUREFIRE-1546.

      In that discussion srdo and marcphilipp have described an approach how CONTAINER-type TestIdentifiers could be handled properly, but that particular problem seems to have been deferred and has seemingly not been picked up again since.

      Workaround

      A workaround is to explicitly reference the displayName or include some other unique string in the name attribute of each @ParameterizedTest as mentioned above. In projects with a large legacy testbase that might not be feasible though, so this issue might prevent the adoption of the feature introduced in SUREFIRE-1546.

      I'm submitting this as a bug because to me the current behaviour of surefire is at least unexpected, but it could also be seen as a request for improvement.

      Attachments

        Issue Links

          Activity

            People

              tibordigana Tibor Digana
              andpab Andreas Pabst
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: