Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.3.2, 2.4.0
-
None
Description
In spark root pom, there is a test-jar execution for all sub-projects including spark-streaming. This will attach an artifact: org.apache.spark:spark-streaming_2.11:test-jar:tests:2.3.2
Also, in streaming pom, there is a shade test configuration, it will attach an artifact: org.apache.spark:spark-streaming_2.11:jar:tests:2.3.2
But two artifacts actually point to the same file: spark-streaming_2.11-2.3.2-tests.jar
So, when deploy spark to nexus using mvn deploy, maven will upload the test.jar twice since it belongs to 2 artifacts. Then deploy fails due to nexus don't allow overrides existing file for non-SNAPSHOT release.
What's more, after checking the spark-streaming, shaded test jar is exactly same to original test jar. It seems we can just delete the related shade config.
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadeTestJar>true</shadeTestJar>
</configuration>
</plugin>
</plugins>
</build>