Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47046

Apache Spark 4.0.0 Dependency Audit and Cleanup

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • Build

    Attachments

      Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        relates to

        Sub-task - The sub-task of the issue SPARK-47018 Upgrade built-in Hive to 2.3.10

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-45869 Revisit and Improve Spark Standalone Cluster

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        1.
        		 Upgrade Derby to 10.15.1.3+ Sub-task Resolved Yang Jie
        2.
        		 Upgrade Derby to 10.16.1.1 Sub-task Resolved Yang Jie
        3.
        		 Add `derby-provided` profile Sub-task Resolved Dongjoon Hyun
        4.
        		 Add `hive-jackson-provided` profile Sub-task Resolved Dongjoon Hyun
        5.
        		 Provide `CodeHaus Jackson` dependencies via a new optional directory Sub-task Resolved Dongjoon Hyun
        6.
        		 Remove Guava from shared classes from IsolatedClientLoader Sub-task Resolved Cheng Pan
        7.
        		 Upgrade `Guava` dependency in `docker-integration-tests` test module Sub-task Resolved Dongjoon Hyun
        8.
        		 Upgrade Guava used by the connect module to 33.1.0-jre Sub-task Resolved BingKun Pan
        9.
        		 Upgrade dropwizard metrics 4.2.21 Sub-task Resolved Yang Jie
        10.
        		 Upgrade ZooKeeper to 3.9.1 Sub-task Resolved Bjørn Jørgensen
        11.
        		 Upgrade `ZooKeeper` to 3.9.2 Sub-task Resolved Dongjoon Hyun
        12.
        		 Remove the special Zookeeper version in the `streaming-kafka-0-10` and `sql-kafka-0-10` modules Sub-task Resolved Yang Jie
        13.
        		 Upgrade Kafka to 3.6.1 Sub-task Resolved Deng Ziming
        14.
        		 Migrating Junit4 to Junit5 Sub-task Resolved Yang Jie
        15.
        		 Upgrade jetty to 9.4.53.v20231009 Sub-task Resolved Yang Jie
        16.
        		 Migrate jetty 9 to jetty 10 Sub-task Resolved HiuFung Kwok
        17.
        		 Upgrade Jetty to 11 Sub-task Resolved HiuFung Kwok
        18.
        		 Remove shaded `protobuf-java` 2.6.1 dependency from `kinesis-asl-assembly` Sub-task Resolved Dongjoon Hyun
        19.
        		 Fix `kvstore` module to have explicit `commons-lang3` test dependency Sub-task Resolved Dongjoon Hyun
        20.
        		 Use `org.seleniumhq.selenium.htmlunit3-driver` instead of `net.sourceforge.htmlunit` Sub-task Resolved Dongjoon Hyun
        21.
        		 Upgrade `aircompressor` to 0.26 Sub-task Resolved Dongjoon Hyun
        22.
        		 Ban non-shaded Hadoop dependencies Sub-task Resolved Dongjoon Hyun
        23.
        		 Add a checkstyle rule to ban `commons-lang` in Java code Sub-task Resolved Dongjoon Hyun
        24.
        		 Add scalastyle and checkstyle rules to ban `FileBackedOutputStream` Sub-task Resolved Dongjoon Hyun
        25.
        		 When there are corrupted files in the local maven repo, retry to skip this cache Sub-task Resolved BingKun Pan
        26.
        		 Add `scalastyle` and `checkstyle` rules to ban `AtomicDoubleArray|CompoundOrdering` Sub-task Resolved Dongjoon Hyun
        27.
        		 Upgrade several Maven plugins to the latest versions Sub-task Resolved Dongjoon Hyun
        28.
        		 Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir Sub-task Resolved Dongjoon Hyun
        29.
        		 Migrate mockito 4 to mockito5 Sub-task Resolved BingKun Pan
        30.
        		 Add `getUriBuilder` to `o.a.s.u.Utils` and use it Sub-task Resolved Dongjoon Hyun
        31.
        		 Upgrade `mockito` to 5.10.0 Sub-task Resolved Dongjoon Hyun
        32.
        		 Upgrade Netty to 4.1.106.Final Sub-task Resolved BingKun Pan
        33.
        		 Add netty-tcnative-boringssl-static dependency Sub-task Resolved Hasnain Lakhani
        34.
        		 Upgrade netty to 4.1.107.Final and netty-tcnative to 2.0.62.Final Sub-task Resolved Dongjoon Hyun
        35.
        		 Upgrade `netty` to 4.1.108.Final and `netty-tcnative` to 2.0.65.Final Sub-task Resolved BingKun Pan
        36.
        		 Upgrade netty to `4.1.109.Final` Sub-task Resolved BingKun Pan
        37.
        		 Upgrade scalatest related dependencies to the 3.2.18 series Sub-task Resolved Yang Jie
        38.
        		 Fix `HistoryServerSuite` to get `getNumJobs` in `eventually` Sub-task Resolved Dongjoon Hyun
        39.
        		 Set `derby.connection.requireAuthentication` to false explicitly in CLIs Sub-task Resolved Dongjoon Hyun
        40.
        		 Upgrade commons-compress.version to 1.24.0 Sub-task Resolved Hyukjin Kwon
        41.
        		 Upgrade commons-compress to 1.25.0 Sub-task Resolved Yang Jie
        42.
        		 Upgrade `commons-compress` to 1.26.0 Sub-task Resolved Dongjoon Hyun
        43.
        		 Upgrade `commons-compress` to 1.26.1` Sub-task Resolved Dongjoon Hyun
        44.
        		 Revert S3A endpoint fixup logic of SPARK-35878 Sub-task Resolved Steve Loughran
        45.
        		 Remove javax-servlet-api exclusion rule for SBT Sub-task Resolved HiuFung Kwok
        46.
        		 Update `SKIP_SPARK_RELEASE_VERSIONS` in Maven CIs Sub-task Resolved Dongjoon Hyun
        47.
        		 Fix `MavenUtilsSuite` to use `MavenUtils.resolveMavenCoordinates` properly Sub-task Resolved Dongjoon Hyun
        48.
        		 Use `spark.jars.ivy` instead `spark.driver.extraJavaOptions` in `DepsTestsSuite` Sub-task Resolved Dongjoon Hyun
        49.
        		 Upgrade Ivy to 2.5.2 Sub-task Resolved Dongjoon Hyun
        50.
        		 Improve `ArtifactSuite` to use unique `MavenCoordinate`s Sub-task Resolved Dongjoon Hyun
        51.
        		 Migrate jersey 2 to jersey 3 Sub-task Resolved Unassigned
        52.
        		 Fix `kafka-0-10-sql` to use `ResetSystemProperties` if `KafkaTestUtils` is used Sub-task Resolved Dongjoon Hyun
        53.
        		 Update K8s `Dockerfile` to include `hive-jackson` directory if exists Sub-task Resolved Dongjoon Hyun
        54.
        		 Fix `make-distribution.sh` to check `jackson-core-asl-1.9.13.jar` existence first Sub-task Resolved Dongjoon Hyun
        55.
        		 Remove redundant scope identifier for `jakarta.servlet-api` and `javax.servlet-api` Sub-task Resolved HiuFung Kwok
        56.
        		 Remove ZOOKEEPER-1844 comment from KafkaTestUtils Sub-task Resolved Dongjoon Hyun
        57.
        		 Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro-*` Sub-task Resolved Dongjoon Hyun
        58.
        		 Upgrade log4j2 to 2.23.0 Sub-task Resolved Unassigned
        59.
        		 Upgrade slf4j to 2.0.12 Sub-task Resolved Yang Jie
        60.
        		 Bump ap-loader 3.0(v8) to support for async-profiler 3.0 Sub-task Resolved Nicholas Jiang
        61.
        		 Upgrade jetty to 11.0.20 Sub-task Resolved Yang Jie
        62.
        		 Upgrade `grpcio*` and `grpc-java` to 1.62 Sub-task Resolved Yang Jie
        63.
        		 Upgrade rocksdbjni to 8.11.3 Sub-task Resolved Yang Jie
        64.
        		 Upgrade RoaringBitmap to 1.0.5 Sub-task Resolved BingKun Pan
        65.
        		 Upgrade `gcs-connector` to 2.2.16 Sub-task Resolved Dongjoon Hyun
        66.
        		 Upgrade `gcs-connector` to 2.2.17 Sub-task Resolved Dongjoon Hyun
        67.
        		 Upgrade `gcs-connector` to 2.2.18 Sub-task Resolved Dongjoon Hyun
        68.
        		 Upgrade `gcs-connector` to 2.2.20 Sub-task Resolved Dongjoon Hyun
        69.
        		 Upgrade jackson to 2.17.0 Sub-task Resolved BingKun Pan
        70.
        		 Upgrade jmock-junit5 to 2.13.1 Sub-task Resolved Yang Jie
        71.
        		 Remove `commons-logging` dependency Sub-task Resolved Dongjoon Hyun
        72.
        		 Remove unused `commons-beanutils` dependency Sub-task Resolved Dongjoon Hyun
        73.
        		 Upgrade `sbt-assembly` to `2.2.0` and `sbt-protoc` to `1.0.7` Sub-task Resolved BingKun Pan
        74.
        		 Upgrade `zstd-jni` to 1.5.6-1 Sub-task Resolved Dongjoon Hyun
        75.
        		 Upgrade `zstd-jni` to 1.5.6-2 Sub-task Resolved BingKun Pan
        76.
        		 Exclude `org.junit.jupiter` and `org.junit.platform` from `jmock-junit5` Sub-task Resolved Yang Jie
        77.
        		 Upgrade `tink` to 1.12.0 Sub-task Resolved BingKun Pan
        78.
        		 Upgrade `gcs-connector` to 2.2.21 and add a note for 3.0.0 Sub-task Resolved Dongjoon Hyun
        79.
        		 Upgrade snappy to 1.1.10.4 Sub-task Resolved Bjørn Jørgensen
        80.
        		 Upgrade snappy to 1.1.10.5 Sub-task Resolved BingKun Pan
        81.
        		 Remove redundant json4s-jackson definition in sql/api POM Sub-task Resolved Cheng Pan
        82.
        		 Upgrade `commons-io` to 2.14.0 Sub-task Resolved BingKun Pan
        83.
        		 Upgrade commons-io to 2.15.0 Sub-task Resolved Yang Jie
        84.
        		 Upgrade commons-io to 2.15.1 Sub-task Resolved BingKun Pan
        85.
        		 Upgrade commons-io to 2.16.0 Sub-task Resolved BingKun Pan
        86.
        		 Upgrade `commons-io` to 2.16.1 Sub-task Resolved Dongjoon Hyun
        87.
        		 Upgrade `commons-codec` to 1.16.1 Sub-task Resolved BingKun Pan
        88.
        		 Bump json4s 4.0.7 Sub-task Resolved Cheng Pan
        89.
        		 Upgrade `postgresql` to 42.7.3 Sub-task Resolved BingKun Pan
        90.
        		 Upgrade `slf4j` to 2.0.13 Sub-task Resolved Dongjoon Hyun
        91.
        		 Upgrade rocksdbjni to 8.11.4 Sub-task Resolved Neil Ramaswamy
        92.
        		 Upgrade commons-text to 1.12.0 Sub-task Resolved Yang Jie
        93.
        		 Upgrade `guava` dependency to `33.1.0-jre` in Docker IT Sub-task Resolved Cheng Pan
        94.
        		 Upgrade `zstd-jni` to 1.5.6-3 Sub-task Resolved BingKun Pan
        95.
        		 Upgrade built-in Hive to 2.3.10 Sub-task Open Unassigned

        Activity

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: