Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
I am running the below commands in databricks.
When I am trying to read a file which is stored in adls using pandas:
pip install adlfs import pandas as pd data = pd.read_parquet("abfss://data.parquet", storage_options= {})
Then I got the below error:
File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py", line 310, in read_parquet return impl.read(path, columns=columns, **kwargs) File "/databricks/python/lib/python3.7/site-packages/pandas/io/parquet.py", line 125, in read path, columns=columns, **kwargs File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line 1573, in read_table ignore_prefixes=ignore_prefixes, File "/databricks/python/lib/python3.7/site-packages/pyarrow/parquet.py", line 1434, in __init__ ignore_prefixes=ignore_prefixes) File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line 667, in dataset return _filesystem_dataset(source, **kwargs) File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line 424, in _filesystem_dataset fs, paths_or_selector = _ensure_single_source(source, filesystem) File "/databricks/python/lib/python3.7/site-packages/pyarrow/dataset.py", line 371, in _ensure_single_source filesystem, path = FileSystem.from_uri(path) File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI: abfss://data.parquet