Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31221

Rebase all dates/timestamps in conversion in Java types

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      Currently, the fromJavaDate(), toJavaDate(), toJavaTimestamp() and fromJavaTimestamp() methods of DateTimeUtils perform rebase only dates before Gregorian cutover date 1582-10-15 assuming that Gregorian calendar has the same behavior in Java 7 and Java 8 API. The assumption is incorrect, in particular, in getting zone offsets, for instance:

      scala> java.time.ZoneId.systemDefault
      res16: java.time.ZoneId = America/Los_Angeles
      scala> java.sql.Timestamp.valueOf("1883-11-10 00:00:00").getTimezoneOffset / 60.0
      warning: there was one deprecation warning; re-run with -deprecation for details
      res17: Double = 8.0
      scala> java.time.ZoneId.of("America/Los_Angeles").getRules.getOffset(java.time.LocalDateTime.parse("1883-11-10T00:00:00"))
      res18: java.time.ZoneOffset = -07:52:58
      

      Java 7 is not accurate, America/Los_Angeles changed time zone shift from

      -7:52:58
      

      to

      -8:00 
      

      The ticket aims to perform rebasing for any dates/timestamps independently from calendar cutover date.

      Attachments

        Activity

          People

            maxgekk Max Gekk
            maxgekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: