Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30097

Adding support for core writers

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Won't Do
    • 3.0.0
    • None
    • SQL, Structured Streaming
    • None
    • 
      

       

    Description

      When using writeStream we always have to use format("xxx") in order to target the selected sink while in r*eadStream* you can use directly .parquet 

      Basically this is to add the support to the core writers for writeStream

      Example:
       

      writeStream
      .outputMode("append")
      .partitionBy("id")
      .options(options)
      .parquet(path)
       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            gschiavon German Schiavon Matteo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment