Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32768

Add Parquet Timestamp output configuration to docs

    XMLWordPrintableJSON

Details

    • Documentation
    • Status: Resolved
    • Minor
    • Resolution: Not A Problem
    • 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 3.0.0
    • None
    • Documentation

    Description

      Spark 2.3.0 added the spark.sql.parquet.outputTimestampType configuration option for controlling the underlying datatype used when writing Timestamp column types into parquet files. This option is helpful for compatibility with external systems that need to read the output from Spark.

      This was never exposed in the documentation. Fix should be applied to docs for both the next 3.x release and 2.4.x release.

      Attachments

        Activity

          People

            Unassigned Unassigned
            drax3d Ron DeFreitas
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: