Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12999

Tez: Vertex creation reduce NN IPCs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.2.0, 1.3.0, 2.0.0, 2.1.0
    • 1.3.0, 2.1.0
    • Tez
    • None

    Description

      Tez vertex building has a decidedly slow path in the code, which is not related to the DAG plan at all.

      The total number of RPC calls is not related to the total number of operators, due to a bug in the DagUtils inner loops.

      	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
	at org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs(Utilities.java:3207)
	at org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs(Utilities.java:3170)
	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:548)
	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1151)
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:388)
	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)

      Attachments

        1. HIVE-12999.1.patch
          1 kB
          Gopal Vijayaraghavan

        Activity

          People

            gopalv Gopal Vijayaraghavan
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: