Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28630

Union fails when column order is different

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.2.3
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      I am trying to union 2 df which has same number of columns and types but in different order. It fails.

            val df1 = sparkSession.sparkContext.parallelize(List(

              (1, 5, true),

              (2, 3, false),

              (4, 4, true)

            )).toDF("id","age", "vaccinated")

       

            val df3 = sparkSession.sparkContext.parallelize(List(

              (1,  true, 6),

              (2, false, 3),

              (3, false, 2)

            )).toDF("id","vaccinated", "age")

      df1.union(df3)

       

      Actual output:

      org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the same number of columns

      Expected output:

       

      It should read schema (column names and/Or types ). I can see sometime you want to ignore column name and just want to do merge based on types. So May be introduce an option whether to use names followed by type or just type while doing merge.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tenstriker nirav patel
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: