Details
Blocker Description
SPARK-20350 introduced a bug BooleanSimplification for null handling. For example, the following case returns a wrong answer.
val schema = StructType.fromDDL("a boolean, b int")
val rows = Seq(Row(null, 1))
val rdd = sparkContext.parallelize(rows)
val df = spark.createDataFrame(rdd, schema)
checkAnswer(df.where("(NOT a) OR a"), Seq.empty)
Attachments
Issue Links
- relates to
-
SPARK-20350 Apply Complementation Laws during boolean expression simplification
-
- Resolved
-
- links to
