Details
Description
Tez vertex building has a decidedly slow path in the code, which is not related to the DAG plan at all.
The total number of RPC calls is not related to the total number of operators, due to a bug in the DagUtils inner loops.
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877) at org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs(Utilities.java:3207) at org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs(Utilities.java:3170) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:548) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1151) at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:388) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)