Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12999

Tez: Vertex creation reduce NN IPCs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.3.0, 2.0.0, 2.1.0
    • Fix Version/s: 1.3.0, 2.1.0
    • Component/s: Tez
    • Labels:
      None

      Description

      Tez vertex building has a decidedly slow path in the code, which is not related to the DAG plan at all.

      The total number of RPC calls is not related to the total number of operators, due to a bug in the DagUtils inner loops.

      	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
      	at org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs(Utilities.java:3207)
      	at org.apache.hadoop.hive.ql.exec.Utilities.createTmpDirs(Utilities.java:3170)
      	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:548)
      	at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1151)
      	at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:388)
      	at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
      

        Attachments

        1. HIVE-12999.1.patch
          1 kB
          Gopal Vijayaraghavan

          Activity

            People

            • Assignee:
              gopalv Gopal Vijayaraghavan
              Reporter:
              gopalv Gopal Vijayaraghavan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: