Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.3.4
-
None
-
None
Description
CommonMergeJoinOperator fails to perform joins on complex keys, e.g.
CREATE TABLE complex_key ( `key` struct<id:bigint, country:string> value int) PARTITIONED BY (date int); SELECT t1.key, t1.value, t2.value FROM complex_key t1 FULL OUTER JOIN complex_key t2 ON (t1.date=20181121 and t2.date =20181122 AND t1.key=t2.key);
This causes a ClassCastException:
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=1) {"key":{"reducesinkkey0":{"id":1,"country":"DK"}},"value":{"_col0":1489}} at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279) ... 22 more Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.WritableComparable at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.compareKeys(CommonMergeJoinOperator.java:543) at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.processKey(CommonMergeJoinOperator.java:516) at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
Because the compareKeys() method tries to cast each key to a WritableComparable but e.g. the StandardStructObjectInspector would return our key field as an Arraylist.
https://github.com/apache/hive/blob/66f97da9de65b1c7151ec57bdf9ada937855bd75/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L590
Proper way to do it would probably be to use the KeyWrapperFactory to convert the keys to something easily comparable?
Attachments
1.
|
Add support for complex type operator in Join with non equality condition | Open | mahesh kumar behera | |
2.
|
Add support for map data type in Common merge join and SMB Join | Open | Unassigned | |
3.
|
Join on union type fails | Open | Unassigned |