Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
-
Using hdp 3.1.4.0-315
Description
Running a query between a big table and a small one, this error appeared:
Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:361) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async Initialization failed. abortRequested=false at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:465) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:399) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:572) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:524) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:342) ... 17 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NegativeArraySizeException at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:103) at org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NegativeArraySizeException at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:271) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) ... 5 more Caused by: java.lang.NegativeArraySizeException at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.<init>(HybridHashTableContainer.java:362) at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer.<init>(HybridHashTableContainer.java:277) at org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:231) ... 9 more
When looking into the stacktrace, this line appeared.
hashPartitions = new HashPartition[numPartitions];
Apparently the int variable numPartition overflows
When this parameter is set hive.auto.convert.join=false the issue disappears
Similar to this Issue https://issues.apache.org/jira/browse/HIVE-12547 but not the same