Description
Old POSIXlt, POSIXct and Date become corrupt in SparkR. For example, see below:
# Non-existent timestamp in hybrid Julian and Gregorian Calendar showDF(createDataFrame(as.data.frame(list(list(POSIXct=as.POSIXct("1582-10-10 00:01:00"), POSIXlt=as.POSIXlt("1582-10-10 00:01:00"))))))
+-------------------+-------------------+ | POSIXct| POSIXlt| +-------------------+-------------------+ |1582-09-30 00:33:08|1582-09-30 00:33:08| +-------------------+-------------------+
See https://docs.google.com/document/d/1an3Mzv6s0naO4mDwGFHJ48gLT--6EliA1GG3kbgBymo/edit?usp=sharing
Note that the results seem wrong from the very first implementation. The cause seems because R side uses Proleptic Gregorian calendar but JVM side is using hybrid Juilian and Gregoiran calendar.
Attachments
Issue Links
- relates to
-
SPARK-26651 Use Proleptic Gregorian calendar
-
- Resolved
-