Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17809

scala.MatchError: BooleanType when casting a struct

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.1
    • None
    • None

    Description

      I have a Dataframe with a struct and I need to rename some fields to lower case before saving it to cassandra.

      It turns out that it's not possible to cast a boolean field of a struct to another boolean field in the renamed struct:

      case class ClassWithBoolean(flag: Boolean)
      case class Parent(cwb: ClassWithBoolean)

      val structCwb: DataType = StructType(Seq(
      StructField("flag", BooleanType, true)
      ))

      Seq(Parent(ClassWithBoolean(true)))
      .toDF
      .withColumn("cwb", $"cwb".cast(structCwb))
      .collect

      scala.MatchError: BooleanType (of class org.apache.spark.sql.types.BooleanType$)

      A workaround is to temporarily cast the field to an Integer and back:

      val structCwbTmp: DataType = StructType(Seq(
      StructField("flag", IntegerType, true)
      ))

      Seq(Parent(ClassWithBoolean(true)))
      .toDF
      .withColumn("cwb", $"cwb".cast(structCwbTmp))
      .withColumn("cwb", $"cwb".cast(structCwb))
      .collect

      Attachments

        Activity

          People

            joshrosen Josh Rosen
            niek Niek Bartholomeus
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: