Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-16442

[Python] The fragments for ORC dataset return base Fragment instead of FileFragment

    XMLWordPrintableJSON

Details

    Description

      From https://github.com/dask/dask/pull/8944#issuecomment-1112620037

      For the ORC file format, we return base Fragment objects instead of the FileFragment subclass (which has more functionality):

      import pyarrow as pa
      import pyarrow.dataset as ds
      from pyarrow import orc
      
      table = pa.table({'a': [1, 2, 3]})
      orc.write_table(table, "test.orc")
      dataset = ds.dataset("test.orc", format="orc")
      fragment = list(dataset.get_fragments())[0]
      
      In [9]: fragment
      Out[9]: <pyarrow._dataset.Fragment at 0x7f97b159beb0>
      
      In [10]: fragment.path
      ---------------------------------------------------------------------------
      AttributeError                            Traceback (most recent call last)
      <ipython-input-10-6477cc57f0cc> in <module>
      ----> 1 fragment.path
      
      AttributeError: 'pyarrow._dataset.Fragment' object has no attribute 'path'
      

      Attachments

        Issue Links

          Activity

            People

              jorisvandenbossche Joris Van den Bossche
              jorisvandenbossche Joris Van den Bossche
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m