Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33772 Build and Run Spark on Java 17
  3. SPARK-36970

Manual disabled format `B` for `date_format` function to compatibility with Java 8 behavior.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      The `date_format` function has some behavioral differences when using JDK 8 and JDK 17 as following:

      the result of select date_format('2018-11-17 13:33:33.333', 'B') in datetime-formatting-invalid.sql with Java 8 is:

      -- !query
      select date_format('2018-11-17 13:33:33.333', 'B')
      -- !query schema
      struct<>
      -- !query output
      java.lang.IllegalArgumentException
      Unknown pattern letter: B
      

      and with Java 17 the result is:

      - datetime-formatting-invalid.sql *** FAILED ***
        datetime-formatting-invalid.sql
        Expected "struct<[]>", but got "struct<[date_format(2018-11-17 13:33:33.333, B):string]>" Schema did not match for query #34
        select date_format('2018-11-17 13:33:33.333', 'B'): -- !query
        select date_format('2018-11-17 13:33:33.333', 'B')
        -- !query schema
        struct<date_format(2018-11-17 13:33:33.333, B):string>
        -- !query output
        in the afternoon (SQLQueryTestSuite.scala:469)
      

       
      From the javadoc we can find that 'B' is used to represent `Pattern letters to output a day period` in Java 17.

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: