Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36310

Fix hasnan() window function in IndexOpsMixin

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • PySpark
    • None

    Description

       

      File "/__w/spark/spark/python/pyspark/pandas/groupby.py", line 1497, in pyspark.pandas.groupby.GroupBy.rank
Failed example:
    df.groupby("a").rank().sort_index()
Exception raised:
...
pyspark.sql.utils.AnalysisException: It is not allowed to use a window function inside an aggregate function. Please use the inner window function in a sub-query.

      As shown above, hasnans() used in "rank" causes "It is not allowed to use a window function inside an aggregate function" exception.

      We shall adjust that.

       

      Attachments

        Issue Links

          relates to

          Umbrella - An overarching type made of sub-tasks SPARK-34849 SPIP: Support pandas API layer on PySpark

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Pull request #33547 [Github] Pull Request #33547 (ueshin)

          Activity

            People

              XinrongM Xinrong Meng
              XinrongM Xinrong Meng
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: