Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47853

jdbc connect to duckdb with error Unrecognized configuration property "path"

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0, 3.4.1, 3.5.0, 3.5.1
    • None
    • Spark Core, SQL
    • None

    Description

      Link issue: _https://github.com/duckdb/duckdb/issues/11651_

      1. reproduce python code 

      ```python

      from pyspark.sql import SparkSession

      if {}name{} == '{}main{}':
          spark = SparkSession.builder \
              .appName("Example Application") \
              .config("spark.master", "local") \
              .config("spark.jars.packages",
                      "io.delta:delta-core_2.12:2.4.0,org.xerial:sqlite-jdbc:3.45.2.0,org.duckdb:duckdb_jdbc:0.9.2") \
              .getOrCreate()

          spark.sql(
              f"""
                  create table default.movies  
                  using jdbc
                  options (url "jdbc:duckdb:database/duckdb.db" , driver "org.duckdb.DuckDBDriver" , dbtable "duckdb.main.test");
                  """
          )

          spark.sql("select * from default.movies").show()

          spark.stop()

      ```

      2. error log

      ```

      16:28:57    Runtime Error in model movies (models/sources/movies.sql)
        An error occurred while calling o40.sql.
        : java.sql.SQLException: Invalid Input Error: Unrecognized configuration property "path"
              at org.duckdb.DuckDBNative.duckdb_jdbc_startup(Native Method)
              at org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:48)
              at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:41)
              at org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:49)
          ```

      3.  relative spark code

      https://github.com/apache/spark/blob/e6b7950f553cff5adc02b8b5195e79cffff3c97c/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala#L64

      Why do we need to replicate the `path` into the JDBC connection?

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            ZhaoWeiNan WeiNan Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: