Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-27315

DynamicPartitionPruningOptimization.generateSemiJoinOperatorPlan throws NPE when hive.tez.dynamic.semijoin.reduction=true

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.3.9, 3.1.2
    • None
    • CBO, Tez
    • None

    Description

      Hive version:

      2.3.9
      3.1.2

       

      Table definition:

      CREATE TABLE `t`(
        `emplid` string COMMENT '', 
        `effdt` string COMMENT '', 
        `mar_status` string COMMENT ''); 

      Query:

      SELECT t1.emplid, t1.effdt, t2.mar_status
      FROM (
       SELECT emplid, max(effdt) AS effdt
       FROM t
       GROUP BY emplid
      ) t1
       LEFT JOIN t t2
       ON t1.emplid = t2.emplid
       AND t1.effdt = t2.effdt; 

      Exception stacktrace:

       

      java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.generateSemiJoinOperatorPlan(DynamicPartitionPruningOptimization.java:463)
              at org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(DynamicPartitionPruningOptimization.java:226)
              at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
              at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
              at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
              at org.apache.hadoop.hive.ql.parse.TezCompiler.runDynamicPartitionPruning(TezCompiler.java:370)
              at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:94)
              at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:140)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11273)
              at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
              at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
              at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              fangbo_worker FangBO
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: