[SPARK-46253] Plan Python data source read using mapInArrow - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 4.0.0
Fix Version/s: 4.0.0
Component/s: PySpark
Labels:
- pull-request-available

Description

Instead of using a regular Python UDTF, we can actually use an arrow UDF and plan the data source read using the mapInArrow operator.

Attachments

Issue Links

links to

GitHub Pull Request #44170

Activity

People

Assignee:: Allison Wang

Reporter:: Allison Wang

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 05/Dec/23 01:13

Updated:: 15/Dec/23 02:54

Resolved:: 12/Dec/23 22:05