Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39543

The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0
    • 3.2.2, 3.3.1, 3.4.0
    • SQL
    • None

    Description

      The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats, example:

      spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").create

      before

      gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.snappy.parquet

      after

      gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.zstd.parquet ...

       

      Attachments

        Activity

          People

            kaifeiYi yikaifei
            kaifeiYi yikaifei
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: