Details
-
Dependency upgrade
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.2.4, 3.3.3, 3.4.0, 3.4.1, 3.5.0
-
None
-
None
Description
updating springboot-dependencies 2.7 to 3.1 breaks spark with springboot compatibilities.
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.7.14</version> <!-- OK 2.7.* works fine, but 3.* ERROR -->
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Internally, everything compile ok, but fail at runtime while creating the SparkSession, with SparkUI:
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.SparkContext.<init>(SparkContext.scala:503) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099) ~[spark-sql_2.13-3.5.0.jar:3.5.0] at scala.Option.getOrElse(Option.scala:201) ~[scala-library-2.13.8.jar:na] at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093) ~[spark-sql_2.13-3.5.0.jar:3.5.0] .... Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[na:na] at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na] at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[na:na] ... 49 common frames omitted
Root cause: major change with springboot 3: 'javax.servlet.Servlet' class is NO more defined in package 'javax.servlet.', but migrated to 'jakarta.servlet.'
The compiled dependency from spark-core to javax-servlet-api is replaced by maven dependencyManagement
It "might be" possible to get rid of maven dependencyManagement or to override several artifact versions, but it is NOT PRACTICAL.
Here is a version that work, much more complex with springboot3 than springboot2.
It requires few of forced versions, and exclusions in maven.
<properties>
<spark.version>3.5.0</spark.version>
<scala.version>2.13</scala.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>3.1.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet-core</artifactId>
<version>2.41</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.9</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>jakarta.servlet</groupId>
<artifactId>jakarta.servlet-api</artifactId>
<version>4.0.4</version>
</dependency>
</dependencies>
there are several examples showing errors and fixed for servlet, log4j, glassfish incompatibilities here as jira attachment, or directly from github: https://github.com/Arnaud-Nauwynck/test-snippets/tree/master/test-springboot-spark
A possible long term solution is to migrate to servlet in package "jakarta.servlet." instead of "javax.servlet." as springboot already did, so there would "not 
" be incompatibilities between very old and much more recent jars.
Any roadmap for this?