Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-18290

[Python] `pyarrow.fs.copy_files` doesn't work if filenames contain special characters

    XMLWordPrintableJSON

Details

    Description

      I can't upload a file called `spam=ham` to a filesystem that emulates an S3 API. I can workaround the issue by renaming the file `spam-ham`. 

      To reproduce, run a filesystem that emulates an S3 API:

      docker run -p 9444:9000 scireum/s3-ninja:latest
      

      Authenticate with the filesystem:

      export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
      export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      

       Then run this Python script:

      import os
      import tempfile
      
      import pyarrow.fs
      
      source = tempfile.mkdtemp()
      file_path = os.path.join(source, "spam=ham")
      open(file_path, "w").close()
      
      filesystem, path = pyarrow.fs.FileSystem.from_uri(
      "s3://bucket?scheme=http&endpoint_override=localhost:9444"
      )
      pyarrow.fs.copy_files(source, path, destination_filesystem=filesystem)
      

      You'll get the error

      OSError: When initiating multiple part upload for key 'spam=ham' in bucket 'bucket': AWS Error [code 22]: The computed request signature does not match the one provided. Check login credentials. (Expected: e70ab9efb620f744abd43d13e8e6846c585a41da543bfb5da67d2fe1ccfd1aaa, Found: 648456e3441dad5a014b2981c71b6e69ccac9732bdcdbe2d363d95105d914340)
      

      This issue is motivated by https://github.com/ray-project/ray/issues/29845.

      Attachments

        Issue Links

          Activity

            People

              apitrou Antoine Pitrou
              bveeramani Balaji Veeramani
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m