Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47946

Nested field's nullable value could be invalid after extracted using GetStructField

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.2
    • None
    • Spark Core, SQL
    • None

    Description

      I've got error when append to table using DataFrameWriterV2.

      The error was occured in TableOutputResolver.checkNullability. This error occurs when the data type of the schema is the same, but the order of the fields is different.

      I found that GetStructField.nullable returns unexpected result.

      override def nullable: Boolean = child.nullable || childSchema(ordinal).nullable 

      Even if nested field has not nullability attribute, it returns true when parent struct has nullability attribute.

      Parent nullability Child nullability Result
      true true true
      true false true
      false true true
      false false false

       

      I think the logic should be changed to get just child's nullability, because both of parent and child should be nullable to be considered nullable.

       

      override def nullable: Boolean = childSchema(ordinal).nullable  

       

       

       

      I want to check current logic is reasonable, or my suggestion can occur other side effect.

      Attachments

        Activity

          People

            Unassigned Unassigned
            bluzy Junyoung Cho
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: