Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Currently when querying via llap external client framework, in case of order by queries -
1. Due to the fact that spark-llap used to wrap actual query in a subquery as mentioned in HIVE-19794
a) We had to detect order by like -
orderByQuery = plan.getQueryProperties().hasOrderBy() || plan.getQueryProperties().hasOuterOrderBy();
Due to this we recently saw an exception like below for one of the queries that did not have an outer order by (It was having an order by in a subquery)
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalStateException: Requested to generate single split. Paths and fileStatuses are expected to be 1. Got paths: 1 fileStatuses: 7
b) Also we had to disable following optimization -
HiveConf.setBoolVar(conf, ConfVars.HIVE_REMOVE_ORDERBY_IN_SUBQUERY, false);
2. By default we have hive.llap.external.splits.order.by.force.single.split=true which forces us to generate single split leading to performance bottleneck.
We should remove ordering support altogether from llap external client repo and let clients handle it at their end.
Attachments
Attachments
Issue Links
- links to