Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22672

Error in tez optimizing SQL statement

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.2.0
    • None
    • hpl/sql
    • None

    Description

      (): FAILED: ConcurrentModificationException null
      java.util.ConcurrentModificationException
      at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
      at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
      at org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationFromSMBJoins(TezCompiler.java:640)
      at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:117)
      at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:138)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11169)
      at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:257)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:485)
      at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1252)
      at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1239)
      at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:191)
      at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:276)
      at org.apache.hive.service.cli.operation.Operation.run(Operation.java:312)
      at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:500)
      at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:487)
      at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
      at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
      at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)

       

      When map traverses, its elements are deleted,The code as follow:

      for (TableScanOperator ts : tsOps) {
      for (ReduceSinkOperator rs : pctx.getRsOpToTsOpMap().keySet()) {
      if (ts == pctx.getRsOpToTsOpMap().get(rs))

      { // match! GenTezUtils.removeBranch(rs); GenTezUtils.removeSemiJoinOperator(pctx, rs, ts); }

      }
      }

      The  function:GenTezUtils.removeSemiJoinOperator(pctx, rs, ts) will delete the eletement from pctx.getRsOpToTsOpMap().

       

       

       

       

       

       

       

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            yinghua_zh yinghua_zh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: