Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31964

Avoid Pandas import for CategoricalDtype with Arrow conversion

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • PySpark
    • None

    Description

      The import for CategoricalDtype changed in Pandas from 0.23 to 1.0 and currently pyspark checks 2 places to import. It would be better check the type as a string and avoid any imports.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-25351 Handle Pandas category type when converting from Python with Arrow

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-31963 Support both pandas 0.23 and 1.0

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #28793 [Github] Pull Request #28793 (BryanCutler)

          Pull request #30114 [Github] Pull Request #30114 (HyukjinKwon)

          Activity

            People

              bryanc Bryan Cutler
              bryanc Bryan Cutler
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: