Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30546

Make interval type more future-proof

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      We've decided to not follow the SQL standard to define the interval type in 3.0. We should try our best to hide intervals from data sources/external catalogs as much as possible, to not leak internals to external systems.

      In Spark 2.4, intervals are exposed in the following places:
      1. The `CalendarIntervalType` is public
      2. `Colum.cast` accepts `CalendarIntervalType` and can cast string to interval.
      3. `DataFrame.collect` can return `CalendarInterval` objects.
      4. UDF can tale `CalendarInterval` as input.
      5. data sources can return IntervalRow directly which may contain `CalendarInterval`.

      In Spark 3.0, we don't want to break Spark 2.4 applications, but we should not expose intervals wider than 2.4. In general, we should avoid leaking intervals to DS v2 and catalog plugins. We should also revert some PostgresSQL specific interval features.

      Attachments

        Issue Links

          Activity

            People

              Qin Yao Kent Yao 2
              Qin Yao Kent Yao 2
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: