Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-25159

Remove support for ordered results in llap external client library

    XMLWordPrintableJSON

Details

    Description

      Currently when querying via llap external client framework, in case of order by queries -

      1. Due to the fact that spark-llap used to wrap actual query in a subquery as mentioned in HIVE-19794

      a) We had to detect order by like -

      orderByQuery = plan.getQueryProperties().hasOrderBy() || plan.getQueryProperties().hasOuterOrderBy();
      

      Due to this we recently saw an exception like below for one of the queries that did not have an outer order by (It was having an order by in a subquery)

      org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
      java.lang.IllegalStateException: Requested to generate single split. Paths and fileStatuses are expected to be 1. Got paths: 1 fileStatuses: 7
      

      b) Also we had to disable following optimization -

      HiveConf.setBoolVar(conf, ConfVars.HIVE_REMOVE_ORDERBY_IN_SUBQUERY, false);
      

      2. By default we have hive.llap.external.splits.order.by.force.single.split=true which forces us to generate single split leading to performance bottleneck.

      We should remove ordering support altogether from llap external client repo and let clients handle it at their end.

      Attachments

        1. HIVE-25159.01.patch
          10 kB
          Shubham Chaurasia

        Issue Links

          Activity

            People

              ShubhamChaurasia Shubham Chaurasia
              ShubhamChaurasia Shubham Chaurasia
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h