Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31868

Wrong results for week-based-year

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Won't Fix
    • 3.0.0, 3.1.0
    • None
    • SQL

    Description

      spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
      spark.sql.legacy.timeParserPolicy	exception
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd')
               > ;
      == Physical Plan ==
      *(1) Project [-28800000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#37]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
      spark.sql.legacy.timeParserPolicy	legacy
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      == Physical Plan ==
      *(1) Project [-31824000000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#53]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
      spark.sql.legacy.timeParserPolicy	exception
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      == Physical Plan ==
      *(1) Project [-28800000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#69]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      1970-01-01 00:00:00
      spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
      spark.sql.legacy.timeParserPolicy	legacy
      spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      == Physical Plan ==
      *(1) Project [-31824000000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#87]
      +- *(1) Scan OneRowRelation[]
      
      
      spark-sql> select to_timestamp('1969-01-01', 'YYYY-MM-dd');
      1968-12-29 00:00:00
      spark-sql> select to_timestamp('1969-01-01', 'yyyy-MM-dd');
      1969-01-01 00:00:00
      spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
      spark.sql.legacy.timeParserPolicy	exception
      spark-sql> select to_timestamp('1969-01-01', 'yyyy-MM-dd');
      1969-01-01 00:00:00
      
      
      == Physical Plan ==
      *(1) Project [-28800000000 AS to_timestamp(2000-01-01, YYYY-MM-dd)#73]
      +- *(1) Scan OneRowRelation[]
      
      spark-sql> select to_timestamp('2000-01-01', 'YYYY-MM-dd');
      1970-01-01 00:00:00
      

      Seems that `-28800000000` is static

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Qin Yao Kent Yao 2
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: