Details
-
Task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When we query using llap external client library, following happens currently -
1. We first need to get splits using LlapBaseInputFormat#getSplits(), this just needs Hive server JDBC url.
2. We then submit those splits to llap and obtain record reader to read data using LlapBaseInputFormat#getRecordReader(). In this step we need following at client side -
- hive.zookeeper.quorum
-hive.llap.daemon.service.hosts
We need to connect to zk to discover llap daemons.
3. Record reader so obtained needs to initiate a TCP connection from client to LLAP Daemon to submit the split.
4. It also needs to initiate another TCP connection from client to output format port in LLAP Daemon to read the data.
In cloud based deployments, we may not be able to make direct connections to Zk registry and LLAP daemons from client as it might run outside vpc.
For 2, we can move daemon discovery logic to get_splits UDF itself which will run in HS2.
For scenarios like 3 and 4, we can expose additional ports on LLAP with additional auth mechanism.
