Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.0, 3.4.1, 3.5.0, 3.5.1
-
None
-
None
Description
Link issue: _https://github.com/duckdb/duckdb/issues/11651_
- reproduce python code
```python
from pyspark.sql import SparkSession
if {}name{} == '{}main{}':
spark = SparkSession.builder \
.appName("Example Application") \
.config("spark.master", "local") \
.config("spark.jars.packages",
"io.delta:delta-core_2.12:2.4.0,org.xerial:sqlite-jdbc:3.45.2.0,org.duckdb:duckdb_jdbc:0.9.2") \
.getOrCreate()
spark.sql(
f"""
create table default.movies
using jdbc
options (url "jdbc:duckdb:database/duckdb.db" , driver "org.duckdb.DuckDBDriver" , dbtable "duckdb.main.test");
"""
)
spark.sql("select * from default.movies").show()
spark.stop()
```
2. error log
```
16:28:57 Runtime Error in model movies (models/sources/movies.sql)
An error occurred while calling o40.sql.
: java.sql.SQLException: Invalid Input Error: Unrecognized configuration property "path"
at org.duckdb.DuckDBNative.duckdb_jdbc_startup(Native Method)
at org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:48)
at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:41)
at org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:49)
```
3. relative spark code
Why do we need to replicate the `path` into the JDBC connection?