Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22216 Improving PySpark/Pandas interoperability
  3. SPARK-20791

Use Apache Arrow to Improve Spark createDataFrame from Pandas.DataFrame

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.1
    • 2.3.0
    • PySpark, SQL
    • None

    Description

      The current code for creating a Spark DataFrame from a Pandas DataFrame uses `to_records` to convert the DataFrame to a list of records and then converts each record to a list. Following this, there are a number of calls to serialize and transfer this data to the JVM. This process is very inefficient and also discards all schema metadata, requiring another pass over the data to infer types.

      Using Apache Arrow, the Pandas DataFrame could be efficiently converted to Arrow data and directly transferred to the JVM to create the Spark DataFrame. The performance will be better and the Pandas schema will also be used so that the correct types will be used.

      Issues with the poor type inference have come up before, causing confusion and frustration with users because it is not clear why it fails or doesn't use the same type from Pandas. Fixing this with Apache Arrow will solve another pain point for Python users and the following JIRAs could be closed:

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-13534 Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-21583 Create a ColumnarBatch with ArrowColumnVectors for row based iteration

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-17804 Pandas dtypes are not correctly inferred by pyspark

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-18178 Importing Pandas Tables with Missing Values

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-11758 Missing Index column while creating a DataFrame from Pandas

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-22417 createDataFrame from a pandas.DataFrame reads datetime64 values as longs

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #19459 [Github] Pull Request #19459 (BryanCutler)

          Pull request #19738 [Github] Pull Request #19738 (BryanCutler)

          Activity

            People

              bryanc Bryan Cutler
              bryanc Bryan Cutler
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: