Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26651

Use Proleptic Gregorian calendar

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Done
    • 2.4.0
    • None
    • SQL

    Description

      Spark 2.4 and previous versions use a hybrid calendar - Julian + Gregorian in date/timestamp parsing, functions and expressions. The ticket aims to switch Spark on Proleptic Gregorian calendar, and use java.time classes introduced in Java 8 for timestamp/date manipulations. One of the purpose of switching on Proleptic Gregorian calendar is to conform to SQL standard which supposes such calendar.

      Release note:

      Spark 3.0 has switched on Proleptic Gregorian calendar in parsing, formatting, and converting dates and timestamps as well as in extracting sub-components like years, days and etc. It uses Java 8 API classes from the java.time packages that based on ISO chronology . Previous versions of Spark performed those operations by using the hybrid calendar (Julian + Gregorian). The changes might impact on the results for dates and timestamps before October 15, 1582 (Gregorian).

      Attachments

        Issue Links

        is related to

        Bug - A problem which impairs or prevents the functions of the product. SPARK-31702 Old POSIXlt, POSIXct and Date become corrupt due to calendar difference

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. SPARK-30688 Spark SQL Unix Timestamp produces incorrect result with unix_timestamp UDF

        • Major - Major loss of function.
        • In Progress

        Bug - A problem which impairs or prevents the functions of the product. SPARK-31404 file source backward compatibility after calendar switch

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-30760 Port `millisToDays` and `daysToMillis` on Java 8 time API

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. SPARK-18381 Wrong date conversion between spark and python for dates before 1583

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. SPARK-30951 Potential data loss for legacy applications after switch to proleptic Gregorian calendar

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        supercedes

        Bug - A problem which impairs or prevents the functions of the product. SPARK-34675 TimeZone inconsistencies when JVM and session timezones are different

        • Major - Major loss of function.
        • Reopened
        links to

        Web Link GitHub Pull Request #23722

        1.
        		 Make typed Timestamp/Date literals consistent to casting Sub-task Resolved Max Gekk Actions
        2.
        		 Use Proleptic Gregorian calendar in casting UTF8String to date/timestamp types Sub-task Resolved Max Gekk Actions
        3.
        		 Caching of DateTimeFormatter Sub-task Resolved Max Gekk Actions
        4.
        		 Get rid of spark.sql.legacy.timeParser.enabled Sub-task Resolved Sean R. Owen Actions
        5.
        		 Cast date/timestamp by Date/TimestampFormatter Sub-task Resolved Max Gekk Actions
        6.
        		 Use java.time API in timestamp/date functions Sub-task Resolved Max Gekk Actions
        7.
        		 CSV schema inferring does not respect spark.sql.legacy.timeParser.enabled Sub-task Resolved Max Gekk Actions
        8.
        		 Support new date/timestamp parser in HadoopFsRelationTest Sub-task Resolved Max Gekk Actions
        9.
        		 Use java.time API for parsing timestamps and dates from JSON Sub-task Resolved Max Gekk Actions
        10.
        		 Use java.time API for parsing timestamps and dates from CSV Sub-task Resolved Max Gekk Actions
        11.
        		 Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings Sub-task Resolved Max Gekk Actions
        12.
        		 Use Proleptic Gregorian calendar in parsing JDBC lower/upper bounds Sub-task Resolved Max Gekk Actions
        13.
        		 Use Timestamp/DateFormatter in CatalogColumnStat Sub-task Resolved Max Gekk Actions
        14.
        		 Benchmark for date/time functions and expressions Sub-task Resolved Max Gekk Actions
        15.
        		 Port DayWeek, DayOfWeek and WeekDay on Proleptic Gregorian calendar Sub-task Resolved Max Gekk Actions
        16.
        		 Get rid of java.util.Calendar in DateTimeUtils Sub-task Resolved Max Gekk Actions
        17.
        		 Use java.time API in timestampAddInterval Sub-task Resolved Max Gekk Actions
        18.
        		 Benchmark the performance regression Sub-task Resolved Max Gekk Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maxgekk Max Gekk Assign to me
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment