Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10232

FixedSizeListArray is incorrectly written/read to/from parquet

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 1.0.1
    • 2.0.0
    • Python
    • None

    Description

      FixedSizeListArray's seem to be either incorrectly written or read to or from Parquet files.

       

      When reading the parquet file, nulls/Nones are returned where the original values should be.

       

      import pyarrow as pa
      import pyarrow.parquet as pq
      import numpy as np
      
      np_data = np.arange(20*4).reshape(20, 4).astype(np.float64)
      pa_data = pa.FixedSizeListArray.from_arrays(np_data.ravel(), 4)
      assert np_data.tolist() == pa_data.tolist()
      
      schema = pa.schema([pa.field("rectangle", pa_data.type)])
      table = pa.table({"rectangle": pa_data}, schema=schema)
      pq.write_table(table, "test.parquet")
      
      in_table = pq.read_table("test.parquet")   
      # rectangle is filled with nulls
      assert in_table.column("rectangle").to_pylist() == pa_data.tolist()
      
      

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sjperkins Simon Perkins
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: