Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45601

stackoverflow when executing rule ExtractWindowExpressions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • 3.2.3
    • 3.3.0
    • SQL
    • None

    Description

      I am encountering stackoverflow errors while executing the following test case. I looked at the source code and it is ExtractWindowExpressions not extracting the window correctly and encountering a dead loop at resolveOperatorsDownWithPruning that is causing it.

       test("agg filter contains window") {
          val src = Seq((1, "b", "c")).toDF("col1", "col2", "col3")
            .withColumn("test",
              expr("count(col1) filter (where min(col1) over(partition by col2 order by col3)>1)"))
          src.show()
        }
      

      Now my question is this kind of in agg filter (window) is the correct usage? Or should I add a check like spark sql and throw an error "It is not allowed to use window functions inside WHERE clause"?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              JacobZheng JacobZheng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: