Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26213

"hive.limit.pushdown.memory.usage" better not be equal to 1.0, otherwise it will raise an error

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.1.2
    • None
    • Documentation
    • None
    • Hive 3.1.2
      os.name=Linux
      os.arch=amd64
      os.version=5.4.0-72-generic
      java.version=1.8.0_162
      java.vendor=Oracle Corporation

    Description

      In hive-default.xml.template

      <property>
        <name>hive.limit.pushdown.memory.usage</name>
        <value>0.1</value>
        <description>
          Expects value between 0.0f and 1.0f.
          The fraction of available memory to be used for buffering rows in Reducesink operator for limit pushdown optimization.
        </description>
      </property>

      Based on the description of hive-default.xml.template, hive.limit.pushdown.memory.usage expects a value between 0.0 and 1.0, setting hive.limit.pushdown.memory.usage to 1.0 means that it expects the available memory of all buffered lines for the limit pushdown optimization, and successfully start hiveserver2.

      Then, call the java api to write a program to establish a jdbc connection as a client to access hive, using JDBCDemo as an example.

      import demo.utils.JDBCUtils;
      public class JDBCDemo{
      public static void main(String[] args) throws Exception
      {   JDBCUtils.init();   JDBCUtils.createDatabase();   JDBCUtils.showDatabases();   JDBCUtils.createTable();   JDBCUtils.showTables();   JDBCUtils.descTable();   JDBCUtils.loadData();   JDBCUtils.selectData();   JDBCUtils.countData();   JDBCUtils.dropDatabase();   JDBCUtils.dropTable();   JDBCUtils.destory(); }
      }
      

      After running the client program, both the client and the hiveserver throw exceptions.

      2022-05-09 19:05:36: Starting HiveServer2
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Hive Session ID = 67a6db8d-f957-4d5d-ac18-28403adab7f3
      Hive Session ID = f9f8772c-5765-4c3e-bcff-ca605c667be7
      OK
      OK
      OK
      OK
      OK
      OK
      OK
      Loading data to table default.emp
      OK
      FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
      liky@ljq1:~/hive_jdbc_test$ ./startJDBC_0.sh 
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/home/liky/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
      Running: drop database if exists hive_jdbc_test
      Running: create database hive_jdbc_test
      Running: show databases
      default
      hive_jdbc_test
      Running: drop table if exists emp
      Running: create table emp(
      empno int,
      ename string,
      job string,
      mgr int,
      hiredate string,
      sal double,
      comm double,
      deptno int
      )
      row format delimited fields terminated by '\t'
      Running: show tables
      emp
      Running: desc emp
      empno   int
      ename   string
      job     string
      mgr     int
      hiredate       string
      sal     double
      comm   double
      deptno int
      Running: load data local inpath '/home/liky/hiveJDBCTestData/data.txt' overwrite into table emp
      Running: select * from emp
      Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Invalid memory usage value 1.0 for hive.limit.pushdown.memory.usage
            at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:380)
            at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:366)
            at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:354)
            at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:293)
            at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:509)
            at demo.utils.JDBCUtils.selectData(JDBCUtils.java:98)
            at demo.test.JDBCDemo.main(JDBCDemo.java:19)

      Setting hive.limit.pushdown.memory.usage to 0.0 has no exception.

      So, setting hive.limit.pushdown.memory.usage to 1.0 is not desirable, hive-default.xml.template is not clear enough for the description of the boundary of the value, it is better to use the interval to indicate the value that is [0.0,1.0).

      Attachments

        Activity

          People

            fujx ECFuzz
            fujx ECFuzz
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: