Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47661

Using from_json with columnNameOfCorruptRecord and ArrayType schema not working

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.5.1
    • None
    • Java API, SQL
    • None

    Description

      When using schemas of type ArrayType using the columnNameOfCorruptRecord parameter is not changing anything... even though the schema does contain the field it is not populated during parsing

      The cause is line 574 in JsonExpressions.scala and onward in running FailureSafeParser where only when the schema is StructType this field is actually handled and taken into account

      Attachments

        Activity

          People

            Unassigned Unassigned
            eylonronen Eylon Ronen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: