XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0, 3.1.0
    • 3.0.0
    • SQL
    • None

    Description

      It's an un-fixable behavior change to fill the gap between SimpleDateFormat and DateTimeFormater and backward-compatibility for different JDKs.

      A lot of effort has been made to prove this at https://github.com/apache/spark/pull/28674

      The existing behavior itself in 2.4 is confusing, e.g.

      spark-sql> select to_timestamp('1', 'w');
      1969-12-28 00:00:00
      spark-sql> select to_timestamp('1', 'u');
      1970-01-05 00:00:00
      

      the 'u' here seems not to go to the Monday of the first week in week-based form or the first day of the year in non-week-based form but go to the Monday of the second week in week-based form.

      spark-sql> select to_timestamp('2020 2020', 'YYYY yyyy');
      2020-01-01 00:00:00
      spark-sql> select to_timestamp('2020 2020', 'yyyy YYYY');
      2019-12-29 00:00:00
      spark-sql> select to_timestamp('2020 2020 1', 'YYYY yyyy w');
      NULL
      spark-sql> select to_timestamp('2020 2020 1', 'yyyy YYYY w');
      2019-12-29 00:00:00
      

      I think we don't need to introduce all the weird behavior from Java.

      Attachments

        Issue Links

          Activity

            People

              Qin Yao Kent Yao 2
              Qin Yao Kent Yao 2
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: