Details
-
Wish
-
Status: Closed
-
Minor
-
Resolution: Won't Fix
-
None
-
None
Description
Currently pyarrow knows how to serialize pandas dataframes but not dask dataframes.
SerializationCallbackError: pyarrow does not know how to serialize objects of type <class 'dask.dataframe.core.DataFrame'>.
Pickling the dask dataframe foregoes the benefits of using pyarrow for the sub dataframes.
Pyarrow support for serializing dask dataframes would allow storing dataframes efficiently in a database instead of a file system (e.g. parquet).