Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33772

Build and Run Spark on Java 17

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • Build

    Description

      Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 17.

      Version Release Date
      Java 17 (LTS) September 2021

      Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along with the release branch cut.

      Supporting new Java version is considered as a new feature which we cannot allow to backport.

      Attachments

        Issue Links

        is related to

        New Feature - A new feature of the product, which has yet to be developed. SPARK-35781 Support Spark on Apple Silicon on macOS natively on Java 17

        • Major - Major loss of function.
        • Resolved

        Umbrella - An overarching type made of sub-tasks SPARK-38396 Improve K8s Integration Tests

        • Major - Major loss of function.
        • Resolved
        is required by

        Improvement - An improvement or enhancement to an existing feature or task. BEAM-12240 Make Beam compatible with next Java LTS version (Java 17)

        • P2 - Default priority. Will be triaged and planned according to community practices.
        • Triage Needed
        relates to

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17177 Java 17 support

        • Major - Major loss of function.
        • Open
        1.
        		 Add Java 17 ea build test to GitHub action Sub-task Resolved William Hyun   Actions
        2.
        		 Update commons-lang3 to 3.11 Sub-task Resolved William Hyun   Actions
        3.
        		 Upgrade checkstyle plugins Sub-task Resolved William Hyun   Actions
        4.
        		 Upgrade ORC to 1.7.1 Sub-task Resolved William Hyun   Actions
        5.
        		 Move Java 17 on GitHub Actions from EA to LTS release Sub-task Resolved Kousuke Saruta   Actions
        6.
        		 Upgrade ZooKeeper to 3.6.2 Sub-task Resolved Yuming Wang   Actions
        7.
        		 Exclude hadoop-client-api transitive dependency from orc-core Sub-task Resolved Kousuke Saruta   Actions
        8.
        		 Change HiveExternalCatalogVersionsSuite to be able to test with Java 17 Sub-task Resolved Kousuke Saruta   Actions
        9.
        		 Fix Invalid value for HourOfAmPm when testing on JDK 14 Sub-task Resolved Yuming Wang   Actions
        10.
        		 Add java.security.jgss/sun.security.krb5 to DEFAULT_MODULE_OPTIONS Sub-task Resolved Yuming Wang   Actions
        11.
        		 Upgrade Scala to 2.12.14 Sub-task Resolved Dongjoon Hyun   Actions
        12.
        		 Upgrade Scala to 2.12.15 Sub-task Resolved Dongjoon Hyun   Actions
        13.
        		 Upgrade ORC to 1.7.0 Sub-task Resolved Dongjoon Hyun   Actions
        14.
        		 Add Daily GitHub Action jobs for Java11/17 Sub-task Resolved Dongjoon Hyun   Actions
        15.
        		 Remove explicit GC options from Scala tests Sub-task Resolved Dongjoon Hyun   Actions
        16.
        		 Add `.java-version` to `.gitignore` and `.rat-excludes` Sub-task Resolved Dongjoon Hyun   Actions
        17.
        		 Add Java17 PySpark daily test coverage Sub-task Resolved Dongjoon Hyun   Actions
        18.
        		 Add Java17 SparkR daily test coverage Sub-task Resolved Dongjoon Hyun   Actions
        19.
        		 Document Java 17 support Sub-task Resolved Dongjoon Hyun   Actions
        20.
        		 Upgrade Maven to 3.8.4 Sub-task Resolved Dongjoon Hyun   Actions
        21.
        		 Upgrade SBT to 1.6.0 Sub-task Resolved Dongjoon Hyun   Actions
        22.
        		 Set file.encoding to UTF-8 for SBT tests Sub-task Resolved Dongjoon Hyun   Actions
        23.
        		 Use static factory methods instead of the deprecated `Byte/Short/Integer/Long` constructors Sub-task Resolved Dongjoon Hyun   Actions
        24.
        		 Disable to_timestamp('366', 'DD') test case Sub-task Resolved Dongjoon Hyun   Actions
        25.
        		 Upgrade Scala 2.13 to 2.13.7 Sub-task Resolved Yang Jie   Actions
        26.
        		 Make dev/mima runs on Java 17 Sub-task Resolved Yang He   Actions
        27.
        		 Make sql/core and dependent modules all UTs pass on Java 17 Sub-task Resolved Yang Jie   Actions
        28.
        		 "SPARK-36464: size returns correct positive number even with over 2GB data" will oom with JDK17 Sub-task Resolved Yang Jie   Actions
        29.
        		 Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior. Sub-task Resolved Yang Jie   Actions
        30.
        		 WholeStageCodegenSparkSubmitSuite Failed with Java 17 Sub-task Resolved Yang Jie   Actions
        31.
        		 `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17 Sub-task Resolved Yang Jie   Actions
        32.
        		 Pass all UTs in `mllib-local` and `mllib` with Java 17 Sub-task Resolved Yang Jie   Actions
        33.
        		 Pass all UTs in `repl` with Java 17 Sub-task Resolved Yang Jie   Actions
        34.
        		 Pass all UTs in `external/avro` with Java 17 Sub-task Resolved Yang Jie   Actions
        35.
        		 Pass all UTs in `sql/hive` with Java 17 Sub-task Resolved Yang Jie   Actions
        36.
        		 Pass all UTs in `yarn` with Java 17 Sub-task Resolved Yang Jie   Actions
        37.
        		 Supplement all micro benchmark results use to Java 17 Sub-task Resolved Yang Jie   Actions
        38.
        		 Upgrade RoaringBitmap to 0.9.23 Sub-task Resolved Yang Jie   Actions
        39.
        		 Add InMemoryColumnarBenchmark and StateStoreBasicOperationsBenchmark Java 11/17 result Sub-task Resolved Yang Jie   Actions
        40.
        		 Try to remove `--add-exports` compile option for Java 17 Sub-task Resolved Yang Jie   Actions
        41.
        		 Add Java 17 support for spark pull request builds Sub-task Closed Unassigned   Actions
        42.
        		 Add a vectorized BLAS implementation Sub-task Resolved Ludovic Henry   Actions
        43.
        		 Support Java 17 in SparkR SystemRequirements Sub-task Resolved Darek

        0%
        Original Estimate - 1h
        Remaining Estimate - 1h
        		Actions
        44.
        		 Support Java 17 in `dev/test-dependencies.sh` Sub-task Closed Unassigned   Actions
        45.
        		 Adapt uses of JDK 17 Internal APIs Sub-task Closed Unassigned   Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            LuciferYang Yang Jie
            dongjoon Dongjoon Hyun
            Votes:
            3 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                Slack

                  Issue deployment