Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33054 Support interval type in PySpark
  3. SPARK-33056

Add mapping between external Python types and CalendarIntervalType

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • PySpark, SQL
    • None

    Description

      We might consider adding support for at least collecting the results of queries containing interval types

      spark.range(1).select(current_timestamp() - current_timestamp()).first()
      

      and possibly a conversion in the opposite direction.

      An obvious candidate would be datetime.timedelta which is mapped to net.razorvine.pickle.objects.TimeDelta, but semantics of these and CalendarInteval differs.

      Attachments

        Activity

          People

            Unassigned Unassigned
            zero323 Maciej Szymkiewicz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: