Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
2.3.0, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
Description
Spark currently allows MapType expressions to be used as input to hash expressions, but I think that this should be prohibited because Spark SQL does not support map equality.
Currently, Spark SQL's map hashcodes are sensitive to the insertion order of map elements:
val a = spark.createDataset(Map(1->1, 2->2) :: Nil) val b = spark.createDataset(Map(2->2, 1->1) :: Nil) // Demonstration of how Scala Map equality is unaffected by insertion order: assert(Map(1->1, 2->2).hashCode() == Map(2->2, 1->1).hashCode()) assert(Map(1->1, 2->2) == Map(2->2, 1->1)) assert(a.first() == b.first()) // In contrast, this will print two different hashcodes: println(Seq(a, b).map(_.selectExpr("hash(*)").first()))
This behavior might be surprising to Scala developers.
I think there's precedence for banning the use of MapType here because we already prohibit MapType in aggregation / joins / equality comparisons (SPARK-9415) and set operations (SPARK-19893).
If we decide that we want this to be an error then it might also be a good idea to add a spark.sql.legacy flag as an escape-hatch to re-enable the old and buggy behavior (in case applications were relying on it in cases where it just so happens to be safe-by-accident (e.g. maps which only have one entry)).
Alternatively, we could support hashing here if we implemented support for comparable map types (SPARK-18134).