Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26211

"hive.server2.webui.max.historic.queries" should be avoided to be set too large, otherwise it will cause blocking

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Blocker
    • Resolution: Unresolved
    • 3.1.2
    • None
    • Clients
    • None
    • Hive 3.1.2
      os.name=Linux
      os.arch=amd64
      os.version=5.4.0-72-generic
      java.version=1.8.0_162
      java.vendor=Oracle Corporation

    Description

      In hive-default.xml.template

      <property>
        <name>hive.server2.webui.max.historic.queries</name>
        <value>25</value>
        <description>The maximum number of past queries to show in HiverSever2 WebUI.</description>
      </property>

      Set hive.server2.webui.max.historic.queries to a relatively large value, take 20000000 as an example, start hiveserver2, it can start hiveserver normally, and logging without exception.

      liky@ljq1:/usr/local/hive/conf$ hiveserver2 
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      2022-05-09 20:03:41: Starting HiveServer2
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Hive Session ID = 0b419706-4026-4a8b-80fe-b79fecbccd4f
      Hive Session ID = 0f9e28d7-0081-4b2f-a743-4093c38c152d

      Next, if you use beeline as a client to connect to hive and send a request for database related operations, for example, if you query all the databases, after successfully executing "show databases", beeline blocks and no other operations can be performed.

      liky@ljq1:/opt/hive$ beeline
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Beeline version 3.1.2 by Apache Hive
      beeline> !connect jdbc:hive2://192.168.1.194:10000/default
      Connecting to jdbc:hive2://192.168.1.194:10000/default
      Enter username for jdbc:hive2://192.168.1.194:10000/default: hive
      Enter password for jdbc:hive2://192.168.1.194:10000/default: *****
      Connected to: Apache Hive (version 3.1.2)
      Driver: Hive JDBC (version 3.1.2)
      Transaction isolation: TRANSACTION_REPEATABLE_READ
      0: jdbc:hive2://192.168.1.194:10000/default> show databases
      . . . . . . . . . . . . . . . . . . . . . .> ;
      INFO : Compiling command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b): show databases
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Semantic Analysis Completed (retrial = false)
      INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
      INFO : Completed compiling command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b); Time taken: 0.393 seconds
      INFO : Concurrency mode is disabled, not creating a lock manager
      INFO : Executing command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b): show databases
      INFO : Starting task [Stage-0:DDL] in serial mode
      INFO : Completed executing command(queryId=liky_20220509202542_15382019-f07b-40ff-840d-1f720df77d8b); Time taken: 0.109 seconds
      INFO : OK
      INFO : Concurrency mode is disabled, not creating a lock manager
      ----------------
      
      database_name
      ----------------
      
      default      
      ----------------
      1 row selected (1.374 seconds)
      

      Also, on the hiveserver side, a runtime null pointer exception is thrown, and the observation log throws no warnings or errors.

      liky@ljq1:/usr/local/hive/conf$ hiveserver2 
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hbase-2.2.2/hbase/maven/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      2022-05-09 20:03:41: Starting HiveServer2
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Hive Session ID = 0b419706-4026-4a8b-80fe-b79fecbccd4f
      Hive Session ID = 0f9e28d7-0081-4b2f-a743-4093c38c152d
      OK
      Exception in thread "HiveServer2-Handler-Pool: Thread-41" java.lang.RuntimeException: java.lang.NullPointerException
            at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89)
            at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
            at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
            at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
            at com.sun.proxy.$Proxy42.close(Unknown Source)
            at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:552)
            at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:241)
            at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService$1.deleteContext(ThriftBinaryCLIService.java:141)
            at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.NullPointerException
            at org.apache.hive.service.cli.operation.OperationManager.getQueryId(OperationManager.java:189)
            at org.apache.hive.service.cli.operation.OperationManager.removeOperation(OperationManager.java:206)
            at org.apache.hive.service.cli.operation.OperationManager.closeOperation(OperationManager.java:287)
            at org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:751)
            at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
            ... 14 more

      If you open another beeline as a client to initiate a database operation request to the hiveserver, the beeline will also block after executing a request and the hiveserver will throw the same exception again.

      If you call java API write a program to establish a jdbc connection as a client to access hive, take JDBCDemo as an example.

      import demo.utils.JDBCUtils;
      public class JDBCDemo{
          public static void main(String[] args) throws Exception {
              JDBCUtils.init();
              JDBCUtils.createDatabase();
              JDBCUtils.showDatabases();
              JDBCUtils.createTable();
              JDBCUtils.showTables();
              JDBCUtils.descTable();
              JDBCUtils.loadData();
              JDBCUtils.selectData();
              JDBCUtils.countData();
              JDBCUtils.dropDatabase();
              JDBCUtils.dropTable();
              JDBCUtils.destory();
          }
      }
      

      Starting the hiveserver and running the written program will block directly after the first database operation request, and the terminal will not throw any hints, in line with beeline's blocking node, the hiveserver throws the same exceptions as before.

      liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Running: drop database if exists hive_jdbc_test
      Running: create database hive_jdbc_test
      ​
      ​
      ​
      ^Cliky@ljq1:~/hive_jdbc_test$
      

      In summary, when the value of hive.server2.webui.max.historic.queries is set to a larger value, it blocks all clients that initiate the request, and the hive server side only throws a runtime null pointer exception, and the observation log throws no related warnings and errors, causing some impact.

      Attachments

        Activity

          People

            fujx ECFuzz
            fujx ECFuzz
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: