Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44251

Potential for incorrect results or NPE when full outer USING join has null key value

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.2, 3.4.1, 3.5.0
    • 3.3.3, 3.4.2, 3.5.0
    • SQL

    Description

      The following query produces incorrect results:

      create or replace temp view v1 as values (1, 2), (null, 7) as (c1, c2);
      create or replace temp view v2 as values (2, 3) as (c1, c2);
      
      select explode(array(c1)) as x
      from v1
      full outer join v2
      using (c1);
      
      -1   <== should be null
      1
      2
      

      The following query fails with a NullPointerException:

      create or replace temp view v1 as values ('1', 2), (null, 7) as (c1, c2);
      create or replace temp view v2 as values ('2', 3) as (c1, c2);
      
      select explode(array(c1)) as x
      from v1
      full outer join v2
      using (c1);
      
      23/06/25 17:06:39 ERROR Executor: Exception in task 0.0 in stage 14.0 (TID 11)
      java.lang.NullPointerException
      	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.generate_doConsume_0$(Unknown Source)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.smj_consumeFullOuterJoinRow_0$(Unknown Source)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.wholestagecodegen_findNextJoinRows_0$(Unknown Source)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
      	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
      	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
      ...
      

      Attachments

        Issue Links

          Activity

            People

              bersprockets Bruce Robbins
              bersprockets Bruce Robbins
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: