Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36077

Support numpy literals as input for pandas-on-Spark APIs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.0
    • None
    • PySpark
    • None

    Description

      Some pandas-on-Spark APIs use PySpark column-related APIs internally, and these column-related APIs don't support numpy literals, thus numpy literals are disallowed as input (e.g. to_replace parameter in replace API). 

      `isin` method has been adjusted in https://github.com/apache/spark/pull/32955 . We ought to adjust other API to support numpy literals.

      Attachments

        Issue Links

          relates to

          Umbrella - An overarching type made of sub-tasks SPARK-34849 SPIP: Support pandas API layer on PySpark

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              Unassigned Unassigned
              XinrongM Xinrong Meng
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: