Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.4.4
-
None
-
None
Description
When calling to_json on a Struct if a key has Null as a value then the key is thrown away.
import pyspark import pyspark.sql.functions as F l = [("a", "foo"), ("b", None)] df = spark.createDataFrame(l, ["id", "data"]) ( df.select(F.struct("*").alias("payload")) .withColumn("payload", F.to_json(F.col("payload")) ).select("payload") .show() )
Produces the following output:
+--------------------+ | payload| +--------------------+ |{"id":"a","data":...| | {"id":"b"}| +--------------------+
The `data` key in the second row has just been silently deleted.
Attachments
Issue Links
- duplicates
-
SPARK-29444 Add configuration to support JacksonGenrator to keep fields with null values
- Resolved