Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47361 Improve JDBC data sources
  3. SPARK-44262

JdbcUtils hardcodes some SQL statements

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.2.0
    • 4.0.0
    • SQL

    Description

      I am currently investigating an integration with the Neo4j JBDC driver and a Spark-based cloud vendor SDK.

       

      This SDK relies on Spark's JdbcUtils to run queries and insert data.

      While JdbcUtils partly delegates to org.apache.spark.sql.jdbc.JdbcDialect for some queries, some others are hardcoded to SQL, see:

      • org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#dropTable
      • org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#getInsertStatement

       

      This works fine for relational databases but breaks for NOSQL stores that do not support SQL translation (like Neo4j).

      Is there a plan to augment the JdbcDialect surface so that it is also responsible for these currently-hardcoded queries?

      Attachments

        Issue Links

          Activity

            People

              fanjia Jia Fan
              fbiville Florent BIVILLE
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: