Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29900

make relation lookup behavior consistent within Spark SQL

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • SQL
    • None

    Description

      Currently, Spark has 2 different relation resolution behaviors:
      1. try to look up temp view first, then try table/persistent view.
      2. try to look up table/persistent view.

      The first behavior is used in SELECT, INSERT and a few commands that support views, like DESC TABLE.

      The second behavior is used in most commands.

      It's confusing to have inconsistent relation resolution behaviors, and the benefit is super small. It's only useful when there are temp view and table with the same name, but users can easily use qualified table name to disambiguate.

      In postgres, the relation resolution behavior is consistent

      cloud0fan=# create schema s1;
      CREATE SCHEMA
      cloud0fan=# SET search_path TO s1;
      SET
      cloud0fan=# create table s1.t (i int);
      CREATE TABLE
      cloud0fan=# insert into s1.t values (1);
      INSERT 0 1
      
      # access table with qualified name
      cloud0fan=# select * from s1.t;
       i 
      ---
       1
      (1 row)
      
      # access table with single name
      cloud0fan=# select * from t;
       i 
      ---
       1
      (1 rows)
      
      # create a temp view with conflicting name
      cloud0fan=# create temp view t as select 2 as i;
      CREATE VIEW
      
      # same as spark, temp view has higher proirity during resolution
      cloud0fan=# select * from t;
       i 
      ---
       2
      (1 row)
      
      # DROP TABLE also resolves temp view first
      cloud0fan=# drop table t;
      ERROR:  "t" is not a table
      
      # DELETE also resolves temp view first
      cloud0fan=# delete from t where i = 0;
      ERROR:  cannot delete from view "t"
      

      Attachments

        1.
        Migrate SHOW TBLPROPERTIES to new framework Sub-task Resolved Terry Kim Actions
        2.
        V2 Command logical plan should use UnresolvedV2Relation for a table Sub-task Resolved Unassigned Actions
        3.
        Commands involved with namespace go thru the new resolution framework. Sub-task Resolved Terry Kim Actions
        4.
        Migrate ALTER TABLE commands to the new resolution framework Sub-task Resolved Unassigned Actions
        5.
        Migrate ALTER VIEW SET/UNSET commands to the new resolution framework Sub-task Resolved Unassigned Actions
        6.
        Introduce UnresolvedTableOrPermanentView for commands that support a table/view but not a temporary view Sub-task Resolved Unassigned Actions
        7.
        Migrate function related commands to new framework Sub-task Resolved Terry Kim Actions
        8.
        Migrate REFRESH TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        9.
        Migrate DESCRIBE COLUMN to new resolution framework Sub-task Resolved Terry Kim Actions
        10.
        Migrate DROP TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        11.
        Migrate ANALYZE TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        12.
        Migrate LOAD DATA to new resolution framework Sub-task Resolved Terry Kim Actions
        13.
        Migrate SHOW CREATE TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        14.
        Support CACHE/UNCACHE TABLE for v2 tables Sub-task Resolved Terry Kim Actions
        15.
        Migrate TRUNCATE TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        16.
        Improve exception messages while handling UnresolvedTable Sub-task Resolved Terry Kim Actions
        17.
        Improve exception messages while handling UnresolvedTableOrView Sub-task Resolved Terry Kim Actions
        18.
        Migrate SHOW COLUMNS to new resolution framework Sub-task Resolved Terry Kim Actions
        19.
        Migrate CACHE TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        20.
        Migrate ALTER TABLE ... RENAME TO to new resolution framework Sub-task Resolved Terry Kim Actions
        21.
        Migrate DROP VIEW to the new resolution framework Sub-task Resolved Terry Kim Actions
        22.
        Migrate SHOW TABLE EXTENDED to new resolution framework Sub-task Resolved Max Gekk Actions
        23.
        Migrate MSCK REPAIR TABLE to the new resolution framework Sub-task Resolved Terry Kim Actions
        24.
        Migrate ALTER VIEW ... SET/UNSET TBLPROPERTIES to new resolution framework Sub-task Resolved Terry Kim Actions
        25.
        Migrate ALTER VIEW ... AS command to the new resolution framework Sub-task Resolved Terry Kim Actions
        26.
        Migrate UNCACHE TABLE to new resolution framework Sub-task Resolved Terry Kim Actions
        27.
        Migrate ALTER TABLE ... RECOVER PARTITIONS to new resolution framework Sub-task Resolved Terry Kim Actions
        28.
        Migrate ALTER TABLE SET [SERDE|SERDEPROPERTIES] command to the new resolution framework Sub-task Resolved Terry Kim Actions
        29.
        Migrate ALTER TABLE ... RENAME TO PARTITION commands to the new resolution framework Sub-task Resolved Terry Kim Actions
        30.
        Migrate ALTER TABLE .. CHANGE COLUMN to new resolution framework Sub-task Resolved Terry Kim Actions
        31.
        Migrate ALTER TABLE ... SET LOCATION to new resolution framework Sub-task Resolved Max Gekk Actions
        32.
        Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to the new resolution framework Sub-task Resolved Terry Kim Actions
        33.
        Migrate ALTER TABLE drop columns command to the new resolution framework Sub-task Resolved Terry Kim Actions
        34.
        Migrate ALTER TABLE commands that alter columns to new framework Sub-task Resolved Unassigned Actions
        35.
        Migrate ALTER TABLE rename column command to the new resolution framework Sub-task Resolved Terry Kim Actions
        36.
        Migrate ALTER TABLE ADD/RENAME COLUMNS command to the new resolution framework Sub-task Resolved Terry Kim Actions
        37.
        Simplify the logical plan names for ALTER TABLE ... COLUMN Sub-task Resolved Wenchen Fan Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            cloud_fan Wenchen Fan

            Dates

              Created:
              Updated:

              Slack

                Issue deployment