Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
Ubuntu LXC (64 bit)
Description
Operator Hooks inserted into Operator.java cause a performance hit even when it is not being used.
For a count(1) query tested with & without the operator hook calls.
with
2013-04-09 07:33:58,920 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 84.07 sec Total MapReduce CPU Time Spent: 1 minutes 24 seconds 70 msec OK 28800991 Time taken: 40.407 seconds, Fetched: 1 row(s)
without
2013-04-09 07:33:02,355 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 68.48 sec ... Total MapReduce CPU Time Spent: 1 minutes 8 seconds 480 msec OK 28800991 Time taken: 35.907 seconds, Fetched: 1 row(s)
The effect is multiplied by the number of operators in the pipeline that has to forward the row - the more operators there are the, the slower the query.
The modification made to test this was
Operator.java
--- ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java +++ ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java @@ -526,16 +526,16 @@ public void process(Object row, int tag) throws HiveException { return; } OperatorHookContext opHookContext = new OperatorHookContext(this, row, tag); - preProcessCounter(); - enterOperatorHooks(opHookContext); + //preProcessCounter(); + //enterOperatorHooks(opHookContext); processOp(row, tag); - exitOperatorHooks(opHookContext); - postProcessCounter(); + //exitOperatorHooks(opHookContext); + //postProcessCounter(); }
Attachments
Attachments
Issue Links
- is blocked by
-
HIVE-4378 Counters hit performance even when not used
- Closed