Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18424

Single Function for Parsing Dates and Times with Formats

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      I've found it quite cumbersome to work with dates thus far in Spark, it can be hard to reason about the timeformat and what type you're working with, for instance:

      say that I have a date in the format

      2017-20-12
      // Y-D-M
      

      In order to parse that into a Date, I have to perform several conversions.

        to_date(
          unix_timestamp(col("date"), dateFormat)
          .cast("timestamp"))
         .alias("date")
      

      I propose simplifying this by adding a to_date function (exists) but adding one that accepts a format for that date. I also propose a to_timestamp function that also supports a format.

      so that you can avoid entirely the above conversion.

      It's also worth mentioning that many other databases support this. For instance, mysql has the STR_TO_DATE function, netezza supports the to_timestamp semantic.

      Attachments

        Issue Links

          Activity

            People

              bill_chambers Bill Chambers
              bill_chambers Bill Chambers
              Herman van Hövell Herman van Hövell
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: