Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-47361

Improve JDBC data sources

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.0.0
    • SQL

    Attachments

      Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        1.
        		 Add guidelines for timestamp mapping in JdbcDialect#getCatalystType Sub-task Resolved Kent Yao  
        2.
        		 SPARK-45561 causes mysql unsigned tinyint overflow Sub-task Resolved Kent Yao  
        3.
        		 Add a general mapping for TIME WITHOUT TIME ZONE to TimestampNTZType Sub-task Resolved Kent Yao  
        4.
        		 Support Oracle TIMESTAMP WITH LOCAL TIME ZONE type Sub-task Resolved Kent Yao  
        5.
        		 Remove timezone limitation for ORACLE TIMESTAMP WITH TIMEZONE Sub-task Resolved Kent Yao  
        6.
        		 Spark: Getting SQLException: Unsupported type -102 reading from Oracle Sub-task Resolved Kent Yao  
        7.
        		 Upgrade oracle jdbc driver to 23.3.0.23.09 Sub-task Resolved BingKun Pan  
        8.
        		 Use gvenzl/oracle-free:23.3-slim to reduce disk usage for docker it Sub-task Resolved Kent Yao  
        9.
        		 Revert SPARK-43049 Sub-task Resolved Kent Yao  
        10.
        		 Fix TimestampNTZ in Postgres Array Sub-task Resolved Kent Yao  
        11.
        		 Error in postgresql when pushing down filter by timestamp_ntz field Sub-task Resolved Pablo Langa Blanco  
        12.
        		 PostgresDialect distinguishes TIMESTAMP from TIMESTAMP_TZ Sub-task Resolved Kent Yao  
        13.
        		 Upgrade `PostgreSQL` JDBC driver to 42.7.2 and docker image to 16.2 Sub-task Resolved Dongjoon Hyun  
        14.
        		 Convert TINYINT catalyst properly in MySQL Dialect Sub-task Resolved Michael Zhang  
        15.
        		 Handle TIMESTAMP and DATETIME in MYSQLDialect Sub-task Resolved Kent Yao  
        16.
        		 Align the mapping ways of other unsigned numeric types with TINYINT UNSGINED in MySQLDialect Sub-task Resolved Kent Yao  
        17.
        		 Upgrade MySQL docker image version to 8.3.0 Sub-task Resolved Kent Yao  
        18.
        		 Improve test coverage for mysql bool synonyms Sub-task Resolved Kent Yao  
        19.
        		 Support TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE Sub-task Closed Unassigned  
        20.
        		 Bump DB2 Docker version to 11.5.8.0 Sub-task Resolved Kent Yao  
        21.
        		 Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1 Sub-task Resolved Kent Yao  
        22.
        		 Mapped TINYINT to ShortType for MsSqlServerDialect Sub-task Resolved Zerui Bao  
        23.
        		 Support TIMESTAMP WITH TIME ZONE for H2Dialect Sub-task Resolved Kent Yao  
        24.
        		 Add DatabricksDialect to handle SQL type conversion Sub-task Resolved Ivan Sadikov  
        25.
        		 Spark wrongly map the BOOLEAN Type to BIT(1) in Snowflake Sub-task Resolved Hayssam Saleh

        0%
        Original Estimate - 2h
        Remaining Estimate - 2h
        26.
        		 Redact JDBC url in errors and logs Sub-task Resolved Max Gekk  
        27.
        		 Port classifyException() in JDBC dialects on error classes Sub-task Resolved Max Gekk  
        28.
        		 Restore backward compatibility of JdbcDialect.classifyException Sub-task Resolved Max Gekk  
        29.
        		 Assign error classes/subclasses to JdbcUtils.classifyException Sub-task Resolved Max Gekk  
        30.
        		 JdbcUtils hardcodes some SQL statements Sub-task Resolved Jia Fan  
        31.
        		 Add some comment for param of JdbcDialect createTable Sub-task Resolved Jia Fan  
        32.
        		 Call convertJavaTimestampToTimeStamp in Array getter Sub-task Resolved Kent Yao  
        33.
        		 Improve the debuggability for docker integration test Sub-task Resolved Kent Yao  
        34.
        		 Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala Sub-task Resolved Nguyen Phan Huy  
        35.
        		 Upgrade docker-java to 3.3.5 Sub-task Resolved Kent Yao  
        36.
        		 upgrade guava from 18.0 to 19.0 for docker-integration-tests Sub-task Resolved Kent Yao  
        37.
        		 Support java.sql.Types.NULL Sub-task Resolved Kent Yao  
        38.
        		 Correctness issue of converting postgres INFINITIES timestamps Sub-task Resolved Kent Yao  
        39.
        		 Java SPI Support for JDBC dialect registering Sub-task Resolved Kent Yao  
        40.
        		 Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialects Sub-task Resolved Kent Yao  
        41.
        		 Save TimestampNTZType as DATETIME in MySQL Sub-task Resolved Kent Yao  
        42.
        		 Read MySQL FLOAT as FloatType to keep consistent with the write side Sub-task Resolved Kent Yao  
        43.
        		 Use MySQL Connector/J for MySQL DB instead of MariaDB Connector/J Sub-task Resolved Kent Yao  
        44.
        		 Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE Sub-task Resolved Kent Yao  
        45.
        		 Respect spark.sql.legacy.charVarcharAsString when casting jdbc type to catalyst type in jdbc Sub-task Resolved Kent Yao  
        46.
        		 Audit MySQL ENUM/SET Types Sub-task Resolved Kent Yao  
        47.
        		 Show a warning message about SQLException if `JDBCTableCatalog.loadTable` fails Sub-task Resolved xleoken  
        48.
        		 Cleanup dead code in MySQLDialect.getCatalystType Sub-task Resolved Kent Yao  
        49.
        		 Document Mapping Spark SQL Data Types from MySQL Sub-task Resolved Kent Yao  
        50.
        		 Document Mapping Spark SQL Data Types to MySQL Sub-task Resolved Kent Yao  
        51.
        		 Fix Postgres bit array issue 'Cannot cast to boolean' Sub-task Resolved Kent Yao  
        52.
        		 Make MySQL data source be able to read bit(n>1) as BinaryType like Postgres Sub-task Resolved Kent Yao  
        53.
        		 Using SMALLINT to Write ShortType to MYSQL Sub-task Resolved Kent Yao  
        54.
        		 Fix NPE when reading mysql bit array as LongType Sub-task Resolved Kent Yao  
        55.
        		 Postgres: Map length unspecified bpchar to StringType Sub-task Resolved Kent Yao  
        56.
        		 Postgres: Support multi dimensional array on the write side Sub-task Resolved Kent Yao  
        57.
        		 Postgres: Add test for complex types Sub-task Resolved Kent Yao  
        58.
        		 Postgres: Document Mapping Spark SQL Data Types from PostgreSQL Sub-task Resolved Kent Yao  
        59.
        		 Special handling of JSON type for MySQL Connector/J 5.x Sub-task Resolved 王俊博  
        60.
        		 Parameterize JDBC Drivers for docker integration tests Sub-task Resolved Kent Yao  
        61.
        		 Postgres: Document Mapping Spark SQL Data Types to PostgreSQL Sub-task Resolved Kent Yao  
        62.
        		 Postgres: Support reading multidimensional arrays Sub-task Resolved Kent Yao  
        63.
        		 Oracle: Support reading AnsiIntervalTypes Sub-task Resolved Kent Yao  
        64.
        		 Handle negative scale for JDBC data sources Sub-task Resolved Kent Yao  
        65.
        		 Truncate exceed decimals with scale first instead of precision from JDBC datasource Sub-task Resolved Kent Yao  
        66.
        		 Replace getArrayDimension with updateExtraColumnMeta Sub-task Resolved Kent Yao  
        67.
        		 Oracle: Document Mapping Spark SQL Data Types from Oracle and add tests Sub-task Resolved Kent Yao  
        68.
        		 Utilize java.sql.RowId.getBytes API directly for UTF8String Sub-task Open Unassigned  
        69.
        		 Oracle: Use VARCHAR2 instead of VARCHAR for VarcharType mapping Sub-task Resolved Kent Yao  
        70.
        		 Oracle: Document Mapping Spark SQL Data Types to Oracle Sub-task Resolved Kent Yao  
        71.
        		 createTableColumnTypes need to be mapped to database types instead of using directly Sub-task Resolved Kent Yao  
        72.
        		 Postgres: Add test and doc for Postgres special numeric values Sub-task Resolved Kent Yao  
        73.
        		 MsSQLServer: Cannot find data type BYTE error Sub-task Resolved Kent Yao  
        74.
        		 MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests Sub-task Resolved Kent Yao  
        75.
        		 MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04 Sub-task Resolved Kent Yao  
        76.
        		 MsSQLServer: Document Mapping Spark SQL Data Types to Microsoft SQL Server Sub-task Resolved Kent Yao  
        77.
        		 Make JdbcUtils.makeGetter handle reading time type as NTZ correctly Sub-task Resolved Kent Yao  
        78.
        		 MsSQLServer: Map datatimeoffset to TimestampType Sub-task Resolved Kent Yao  
        79.
        		 MsSQLServer: Fix the scope of spark.sql.legacy.mssqlserver.numericMapping.enabled Sub-task Resolved Kent Yao  
        80.
        		 SQLServer does not support LIKE operator in binary comparison Sub-task Resolved Stefan Bukorovic  
        81.
        		 MsSQLServer: upgrade mssql.jdbc.version to 12.6.1.jre11 Sub-task Resolved Kent Yao  

        Activity

          People

            yao Kent Yao
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 2h
                2h
                Remaining:
                Remaining Estimate - 2h
                2h
                Logged:
                Time Spent - Not Specified
                Not Specified