Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39017

Change Java8 datetime support to configurable

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Brainstorming
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.1
    • None
    • SQL

    Description

      Background:

      In the Spark 3.1.0 there is an improvement introduced to enable Java 8 datetime API by default.

      It stops user set this configuration, spark.sql.datetime.java8API.enabled, in command line or configuration file when using Spark SQL shell or Spark Thrift Server.

      The only way to set it is in the SQL Session using SET command like:

      spark-sql> SET spark.sql.datetime.java8API.enabled=false 

      There are a few issues related to this improvement 

      There is a solution in 3.2.0 for it by having LocalDateConverter uses DateConverter and in the DateConverter it handles both LocalDate and Date types.

       

      Discussion:

      I think we should give back the ability to user to set this option in configuration file and command line. Both of the improvement defeats the reason why we have a java8API option configurable.

       

      Please advise

      Thanks

       

       

       

       

       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            BenWang Weicheng Wang

            Dates

              Created:
              Updated:

              Slack

                Issue deployment