Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16216

CSV data source does not write date and timestamp correctly

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.0.0
    • 2.0.1, 2.1.0
    • SQL

    Description

      Currently, CSV data source write DateType and TimestampType as below:

      +----------------+
|            date|
+----------------+
|1440637200000000|
|1414459800000000|
|1454040000000000|
+----------------+

      It would be nicer if it write dates and timestamps as a formatted string just like JSON data sources.

      Also, CSV data source currently supports dateFormat option to read dates and timestamps in a custom format. It might be better if this option can be applied in writing as well.

      Attachments

        Issue Links

        duplicates

        Bug - A problem which impairs or prevents the functions of the product. SPARK-16597 DataFrame DateType is written as an int(Days since epoch) by csv writer

        • Major - Major loss of function.
        • Closed
        is duplicated by

        Bug - A problem which impairs or prevents the functions of the product. SPARK-17066 dateFormat should be used when writing dataframes as csv files

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SPARK-17923 dateFormat unexpected kwarg to df.write.csv

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        links to

        Pull request #13912 [Github] Pull Request #13912 (HyukjinKwon)

        Pull request #14279 [Github] Pull Request #14279 (HyukjinKwon)

        Pull request #14799 [Github] Pull Request #14799 (HyukjinKwon)

        Pull request #14829 [Github] Pull Request #14829 (HyukjinKwon)

        Pull request #14840 [Github] Pull Request #14840 (HyukjinKwon)

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gurwls223 Hyukjin Kwon
            gurwls223 Hyukjin Kwon
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment