Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.5.0
Description
When create a StructType from a Python dictionary you use StructType.fromJson or in scala DataType.fromJson
To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.
schema = {
"name": "name", "type": "string"
}
StructField.fromJson(schema)
Python Error:
from pyspark.sql.types import StructField schema = { "name": "c1", "type": "string" } StructField.fromJson(schema) >> Traceback (most recent call last): File "code.py", line 90, in runcode exec(code, self.locals) File "<input>", line 1, in <module> File "pyspark/sql/types.py", line 583, in fromJson json["nullable"], KeyError: 'nullable'
Scala Error:
val schema = """ |{ | "type": "struct", | "fields": [ | { | "name": "c1", | "type": "string", | "nullable": false | } | ] |} |""".stripMargin DataType.fromJson(schema) >> Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field. java.lang.IllegalArgumentException: Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field. at org.apache.spark.sql.types.DataType$.parseStructField(DataType.scala:268) at org.apache.spark.sql.types.DataType$.$anonfun$parseDataType$1(DataType.scala:225)
