Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
3.1.1
-
None
-
None
-
Hive-3.1
Description
Steps To Repro:
// MySQL Table CREATE TABLE `visitors` ( `id` bigint(20) unsigned NOT NULL, `date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ) // hive table CREATE EXTERNAL TABLE `hive_visitors`( `col1` bigint COMMENT 'from deserializer', `col2` timestamp COMMENT 'from deserializer') ROW FORMAT SERDE 'org.apache.hive.storage.jdbc.JdbcSerDe' STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' WITH SERDEPROPERTIES ( 'serialization.format'='1') TBLPROPERTIES ( 'bucketing_version'='2', 'hive.sql.database.type'='MYSQL', 'hive.sql.dbcp.maxActive'='1', 'hive.sql.dbcp.password'='hive', 'hive.sql.dbcp.username'='hive', 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver', 'hive.sql.jdbc.url'='jdbc:mysql://hostname/test', 'hive.sql.table'='visitors', 'transient_lastDdlTime'='1554910389') Query: select * from hive_visitors ; Exception: 2019-10-16T04:04:39,483 WARN [HiveServer2-Handler-Pool: Thread-71]: thrift.ThriftCLIService (:()) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.NullPointerException at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:478) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112] at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112] at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.1.1.3.1.4.0-315.jar:?] at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at com.sun.proxy.$Proxy42.fetchResults(Unknown Source) ~[?:?] at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Caused by: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] ... 24 more Caused by: java.lang.NullPointerException at org.apache.hive.storage.jdbc.JdbcSerDe.deserialize(JdbcSerDe.java:164) ~[hive-jdbc-handler-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:583) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
setting fetch.task.conversion=none avoids NPE and yield correct results, the column key lookup failed because of hashcode mismatch and value set to NULL.
jdbc Iterator has right hive column names (col1,col2), the MapWritable has Text(col0) as key
while the JdbcSerDe org.apache.hive.storage.jdbc.JdbcSerDe#hiveColumnNames has incorrect mapping , the columnKey created at org/apache/hive/storage/jdbc/JdbcSerDe.java:157 is Text(id) where id is column name for mysql not hive, opened another Jira HIVE-22353 for this invalid reference issue.