Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3850

[Python] Support MapType and StructType for enhanced PySpark integration

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.11.1
    • 2.0.0
    • Python
    • None

    Description

      It would be great to support MapType and (nested) StructType in Arrow so that PySpark can make use of it.

       
      Quite often as in my use-case in Hive table cells are also complex types saved. Currently it's not possible to user the new pandas_udf decorator which internally uses Arrow to generate a UDF for columns with complex types.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-24554 Add MapType Support for Arrow in PySpark

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6904 [Python] Implement MapArray and MapType

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              FlorianWilhelm Florian Wilhelm
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: