Description
I am encountering stackoverflow errors while executing the following test case. I looked at the source code and it is ExtractWindowExpressions not extracting the window correctly and encountering a dead loop at resolveOperatorsDownWithPruning that is causing it.
test("agg filter contains window") { val src = Seq((1, "b", "c")).toDF("col1", "col2", "col3") .withColumn("test", expr("count(col1) filter (where min(col1) over(partition by col2 order by col3)>1)")) src.show() }
Now my question is this kind of in agg filter (window) is the correct usage? Or should I add a check like spark sql and throw an error "It is not allowed to use window functions inside WHERE clause"?
Attachments
Issue Links
- is fixed by
-
SPARK-38666 Missing aggregate filter checks
- Resolved