Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27943

NPE in VectorMapJoinCommonOperator.setUpHashTable when running query with join on date

    XMLWordPrintableJSON

Details

    Description

      The error can be reproduced on current master using the following steps.

      set hive.auto.convert.join=true;

CREATE TABLE person (fname string, birthDate date);
INSERT INTO person VALUES ('Victor', '2023-11-27'), ('Alexandre', '2023-11-28');

SELECT * FROM person p1 INNER JOIN person p2 ON p1.birthDate=p2.birthDate;

      The stacktrace is shown below.

      2023-12-07T07:06:39,744 ERROR [TezTR-592758_1_2_1_0_0] tez.TezProcessor: Failed initializeAndRunProcessor
java.lang.RuntimeException: Map operator initialization failed
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:351) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:292) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:276) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_261]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_261]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39) ~[tez-runtime-internals-0.10.2.jar:0.10.2]
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) ~[tez-common-0.10.2.jar:0.10.2]
        at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118) ~[hive-llap-server-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_261]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_261]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_261]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.setUpHashTable(VectorMapJoinCommonOperator.java:673) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator.completeInitializationOp(VectorMapJoinCommonOperator.java:634) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:450) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:382) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:549) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:503) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:369) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:332) ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
        ... 16 more

      This is probably caused by HIVE-23852 which added the DATE type as a possible key for the HashTable.

      Attachments

        1. HIVE-27943_plan.txt
          4 kB
          Stamatis Zampetakis

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-23852 Natively support Date type in ReduceSink operator

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #4929

          Activity

            People

              zabetak Stamatis Zampetakis
              zabetak Stamatis Zampetakis
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: