Details
Description
When saving results through spark dataframe on latest 3.0.1-snapshot compiled against hadoop-3.2 with the following specs
--conf spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
--conf spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
--conf spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
--conf spark.hadoop.fs.s3a.committer.name=partitioned
--conf spark.hadoop.fs.s3a.committer.staging.conflict-mode=replace
we are unable to save the file with whitespace character in the path. It works fine without.
I was looking into the recent commits with regards to qualifying the path, but couldn't find anything obvious. Is this a known bug?
When saving results through spark dataframe on latest 3.0.1-snapshot compiled against hadoop-3.2 with the following specs
--conf spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
--conf spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter
--conf spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
--conf spark.hadoop.fs.s3a.committer.name=partitioned
--conf spark.hadoop.fs.s3a.committer.staging.conflict-mode=replace
we are unable to save the file with whitespace character in the path. It works fine without.
I was looking into the recent commits with regards to qualifying the path, but couldn't find anything obvious. Is this a known bug?
Attachments
Attachments
Issue Links
- links to