Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-15285

[C++] write_dataset with delete_matching occasionally fails with "Path does not exist"

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • C++

    Description

      The reproducer in ARROW-15265, once the bug there is fixed, now occasionally fails with this:

      Traceback (most recent call last):
  File "/home/lidavidm/Code/upstream/arrow-15265/python/test.py", line 37, in <module>
    ds.write_dataset(
  File "/home/lidavidm/Code/upstream/arrow-15265/python/pyarrow/dataset.py", line 931, in write_dataset
    _filesystemdataset_write(
  File "pyarrow/_dataset.pyx", line 2658, in pyarrow._dataset._filesystemdataset_write
    check_status(CFileSystemDataset.Write(c_options, c_scanner))
  File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
    raise IOError(message)
OSError: Path does not exist 'my-bucket/test8.parquet/col1=c'

      The path is different each time it fails (i.e. it's not deterministic). This is relatively rare (2 out of 100 runs when I checked just now)

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. ARROW-15265 [C++][Python][Dataset] write_dataset with delete_matching hangs when the number of partitions is too large

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              lidavidm David Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: