Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Information Provided
-
4.0.1
-
None
-
None
-
Ubuntu 20.04
Description
arrow-rs seems unable to load Feather v2 files created by pyarrow (and pandas), while it can read Feather v2 created by itself.
More info at:
https://github.com/apache/arrow-rs/issues/286
Any idea what is missing in the Rust implementation (missing part of the spec?)?
More details: in both files, I am getting the following:
Reading Utf8
field_node: FieldNode { length: 7, null_count: 0 }
offset buffer: Buffer { offset: 200, length: 55 }
offsets: [32, 0, 407708164, 545407072, 8388608, 67108864, 134217728, 201326592]
values buffer: Buffer { offset: 256, length: 51 }
offsets[0] != 0 indicates a problem: offsets are expected to start from zero on any array with offsets.
offsets[i+1] < offsets[i+1] for some i, which indicates a problem: offsets are expected to be monotonically increasing
I do not have a root cause yet, these are just observations.
https://github.com/apache/arrow-rs/issues/286#issuecomment-839524898
In the attachment the following files can be found.
{{}}
test_pandas.feather: Original Feather file test_arrow.feather: loading test_pandas.feather with pyarrow and saving with pyarrow: df_pa = pa.feather.read_feather('test_pandas.feather') test_polars.feather: Loading test_pandas.feather with pyarrow and saving with polars (only this one can be read by arrow-rs) test_pandas_from_polars.feather: Loading test_polars.feather with polars and using the to_pandas option.
[^test_feather_file.zip]