Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Incomplete
-
2.3.1
-
None
Description
Currently I am able to chain .conf.set() settings in the spark session API. I would also like to be able to chain config options in a similar manner in pyspark.sql.conf.
The same issue is present in the scala SQL API so this might also need to be changed for consistency.
See https://github.com/apache/spark/blob/master/python/pyspark/conf.py#L130 vsĀ https://github.com/apache/spark/blob/master/python/pyspark/sql/conf.py#L41