Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.2
-
None
-
None
Description
I've got error when append to table using DataFrameWriterV2.
The error was occured in TableOutputResolver.checkNullability. This error occurs when the data type of the schema is the same, but the order of the fields is different.
I found that GetStructField.nullable returns unexpected result.
override def nullable: Boolean = child.nullable || childSchema(ordinal).nullable
Even if nested field has not nullability attribute, it returns true when parent struct has nullability attribute.
Parent nullability | Child nullability | Result |
---|---|---|
true | true | true |
true | false | true |
false | true | true |
false | false | false |
I think the logic should be changed to get just child's nullability, because both of parent and child should be nullable to be considered nullable.
override def nullable: Boolean = childSchema(ordinal).nullable
I want to check current logic is reasonable, or my suggestion can occur other side effect.