Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-28207

NullPointerException is thrown when checking column uniqueness

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • 4.1.0
    • CBO

    Description

      In some cases, we skip checking null. For example, the last statement in the following set of queries fails with NPE.

      CREATE TABLE `store_sales` (`ss_item_sk` bigint);
      CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
      CREATE TABLE `item` (`i_item_sk` bigint);
      ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`) DISABLE NOVALIDATE RELY;
      ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE NOVALIDATE RELY;
      ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`) REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;
      
      EXPLAIN
      SELECT i_item_sk
      FROM store_sales, household_demographics, item
      WHERE ss_item_sk = i_item_sk

      The NPE happens with HiveJoinConstraintsRule in the above case.

      org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: NullPointerException null
           at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:376) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:214) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:270) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.operation.Operation.run(Operation.java:286) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:557) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:542) ~[hive-service-4.0.0.jar:4.0.0]
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
           at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
           at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-4.0.0.jar:4.0.0]
           at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_275]
           at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_275]
           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
           at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-4.0.0.jar:4.0.0]
           at com.sun.proxy.$Proxy42.executeStatementAsync(Unknown Source) ~[?:?]
           at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:652) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1670) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1650) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-4.0.0.jar:4.0.0]
           at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[hive-exec-4.0.0.jar:4.0.0]
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]
           at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
      Caused by: java.lang.NullPointerException
           at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveJoinConstraintsRule.onMatch(HiveJoinConstraintsRule.java:277) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2446) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2405) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:1943) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1686) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1567) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1319) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:570) ~[hive-exec-4.0.0.jar:4.0.0]
           at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13149) ~[hive-exec-4.0.0.jar:4.0.0] 

      Attachments

        Issue Links

          Activity

            People

              okumin Shohei Okumiya
              okumin Shohei Okumiya
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: