[HIVE-20962] CommonMergeJoinOperator cannot join on complex keys - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 2.3.4
Fix Version/s: None
Component/s: Query Processor
Labels:
None

Description

CommonMergeJoinOperator fails to perform joins on complex keys, e.g.

CREATE TABLE complex_key (
`key` struct<id:bigint, country:string> 
value int)
PARTITIONED BY (date int);

SELECT t1.key, t1.value, t2.value FROM
complex_key t1
FULL OUTER JOIN
complex_key t2
ON (t1.date=20181121 and t2.date =20181122 AND t1.key=t2.key);

This causes a ClassCastException:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=1) {"key":{"reducesinkkey0":{"id":1,"country":"DK"}},"value":{"_col0":1489}}
	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
	... 22 more
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.WritableComparable
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.compareKeys(CommonMergeJoinOperator.java:543)
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.processKey(CommonMergeJoinOperator.java:516)
	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:212)
	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)

Because the compareKeys() method tries to cast each key to a WritableComparable but e.g. the StandardStructObjectInspector would return our key field as an Arraylist.
https://github.com/apache/hive/blob/66f97da9de65b1c7151ec57bdf9ada937855bd75/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L590

Proper way to do it would probably be to use the KeyWrapperFactory to convert the keys to something easily comparable?