Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31085 Amend Spark's Semantic Versioning Policy
  3. SPARK-31136

Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Blocker
    • Resolution: Won't Do
    • 3.0.0
    • None
    • SQL
    • None

    Description

      We need to consider the behavior change of SPARK-30098 .
      This is a placeholder to keep the discussion and the final decision.

      `CREATE TABLE` syntax changes its behavior silently.

      The following is one example of the breaking the existing user data pipelines.
      Apache Spark 2.4.5

      spark-sql> CREATE TABLE t(a STRING);

spark-sql> LOAD DATA INPATH '/usr/local/spark/README.md' INTO TABLE t;

spark-sql> SELECT * FROM t LIMIT 1;
# Apache Spark
Time taken: 2.05 seconds, Fetched 1 row(s)

      spark-sql> CREATE TABLE t(a CHAR(3));

spark-sql> INSERT INTO TABLE t SELECT 'a ';

spark-sql> SELECT a, length(a) FROM t;
a  	3

      Apache Spark 3.0.0-preview2

      spark-sql> CREATE TABLE t(a STRING);

spark-sql> LOAD DATA INPATH '/usr/local/spark/README.md' INTO TABLE t;
Error in query: LOAD DATA is not supported for datasource tables: `default`.`t`;

      spark-sql> CREATE TABLE t(a CHAR(3));

spark-sql> INSERT INTO TABLE t SELECT 'a ';

spark-sql> SELECT a, length(a) FROM t;
a 	2

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-30098 Add a configuration to use default datasource as provider for CREATE TABLE command

          • Major - Major loss of function.
          • Resolved
          is cloned by

          Sub-task - The sub-task of the issue SPARK-31707 Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          relates to

          Sub-task - The sub-task of the issue SPARK-31147 forbid CHAR type in non-Hive-Serde tables

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-31181 Remove the default value assumption on CREATE TABLE test cases

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Documentation - Documentation or Website SPARK-31133 fix sql ref doc for DML

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #28517 [Github] Pull Request #28517 (HeartSaVioR)

          Pull request #28517 [Github] Pull Request #28517 (HeartSaVioR)

          Web Link GitHub Pull Request #27894

          Activity

            People

              Unassigned Unassigned
              dongjoon Dongjoon Hyun
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: