Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
Impala 4.4.0
-
None
-
None
-
ghx-label-7
Description
During dataload in testdata/bin/create-load-data.sh, we run a couple Hive statements to warm up Hive. These produce hundreds of lines of output. This should be diverted to a log file to avoid the noise. Using the run-step function is the standard way to do that.
20:37:29 Running warm up Hive statements 20:37:30 SLF4J: Class path contains multiple SLF4J bindings. 20:37:30 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:30 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:30 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 20:37:31 SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 20:37:32 SLF4J: Class path contains multiple SLF4J bindings. 20:37:32 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:32 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:32 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 20:37:32 SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 20:37:32 Connecting to jdbc:hive2://localhost:11050/default; 20:37:32 Connected to: Apache Hive (version 3.1.3000.7.2.18.0-369) 20:37:32 Driver: Hive JDBC (version 3.1.3000.7.2.18.0-369) 20:37:32 Transaction isolation: TRANSACTION_REPEATABLE_READ 20:37:34 INFO : Compiling command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326): create database if not exists functional 20:37:34 INFO : Semantic Analysis Completed (retrial = false) 20:37:34 INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) 20:37:34 INFO : Completed compiling command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326); Time taken: 1.139 seconds 20:37:34 INFO : Executing command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326): create database if not exists functional 20:37:34 INFO : Starting task [Stage-0:DDL] in serial mode 20:37:34 INFO : Completed executing command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326); Time taken: 0.226 seconds 20:37:34 INFO : OK 20:37:34 No rows affected (1.572 seconds) 20:37:34 Beeline version 3.1.3000.7.2.18.0-369 by Apache Hive 20:37:34 Closing: 0: jdbc:hive2://localhost:11050/default; 20:37:35 SLF4J: Class path contains multiple SLF4J bindings. 20:37:35 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:35 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:35 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 20:37:35 SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 20:37:36 SLF4J: Class path contains multiple SLF4J bindings. 20:37:36 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:36 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:36 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 20:37:36 SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 20:37:37 Connecting to jdbc:hive2://localhost:11050/default; 20:37:37 Connected to: Apache Hive (version 3.1.3000.7.2.18.0-369) 20:37:37 Driver: Hive JDBC (version 3.1.3000.7.2.18.0-369) 20:37:37 Transaction isolation: TRANSACTION_REPEATABLE_READ 20:37:37 INFO : Compiling command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7): create table if not exists hive_warm_up_tbl (i int) 20:37:37 INFO : Semantic Analysis Completed (retrial = false) 20:37:37 INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) 20:37:37 INFO : Completed compiling command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7); Time taken: 0.075 seconds 20:37:37 INFO : Executing command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7): create table if not exists hive_warm_up_tbl (i int) 20:37:37 INFO : Starting task [Stage-0:DDL] in serial mode 20:37:37 INFO : Completed executing command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7); Time taken: 0.101 seconds 20:37:37 INFO : OK 20:37:37 No rows affected (0.253 seconds) 20:37:37 Beeline version 3.1.3000.7.2.18.0-369 by Apache Hive 20:37:37 Closing: 0: jdbc:hive2://localhost:11050/default; 20:37:38 SLF4J: Class path contains multiple SLF4J bindings. 20:37:38 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:38 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:38 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 20:37:38 SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 20:37:40 SLF4J: Class path contains multiple SLF4J bindings. 20:37:40 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:40 SLF4J: Found binding in [jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] 20:37:40 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 20:37:40 SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 20:37:40 Connecting to jdbc:hive2://localhost:11050/default; 20:37:40 Connected to: Apache Hive (version 3.1.3000.7.2.18.0-369) 20:37:40 Driver: Hive JDBC (version 3.1.3000.7.2.18.0-369) 20:37:40 Transaction isolation: TRANSACTION_REPEATABLE_READ 20:37:50 INFO : Compiling command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f): insert overwrite table hive_warm_up_tbl values (1) 20:37:50 INFO : Semantic Analysis Completed (retrial = false) 20:37:50 INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null)], properties:null) 20:37:50 INFO : Completed compiling command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f); Time taken: 0.949 seconds 20:37:50 INFO : Executing command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f): insert overwrite table hive_warm_up_tbl values (1) 20:37:50 INFO : Query ID = jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f 20:37:50 INFO : Total jobs = 3 20:37:50 INFO : Launching Job 1 out of 3 20:37:50 INFO : Starting task [Stage-1:MAPRED] in serial mode 20:37:50 INFO : Subscribed to counters: [] for queryId: jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f 20:37:50 INFO : Tez session hasn't been created yet. Opening session 20:37:50 INFO : Dag name: insert overwrite table hive_warm_up_tb...(1) (Stage-1) 20:37:50 INFO : HS2 Host: [impala-ec2-centos79-m6i-4xlarge-xldisk-0293.vpc.cloudera.com], Query ID: [jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f], Dag ID: [dag_1702614944517_0001_1], DAG Session ID: [application_1702614944517_0001] 20:37:50 INFO : Status: Running (Executing on YARN cluster with App id application_1702614944517_0001) 20:37:50 20:37:51 [2K---------------------------------------------------------------------------------------------- 20:37:51 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 20:37:51 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:51 [2KMap 1 container INITED 1 0 0 1 0 0 20:37:51 [2K---------------------------------------------------------------------------------------------- 20:37:51 [2K[31;1mVERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 2.74 s 20:37:51 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:52 [7A[2K---------------------------------------------------------------------------------------------- 20:37:52 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 20:37:52 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:52 [2KMap 1 container INITED 1 0 0 1 0 0 20:37:52 [2K---------------------------------------------------------------------------------------------- 20:37:52 [2K[31;1mVERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 3.74 s 20:37:52 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:53 [7A[2K---------------------------------------------------------------------------------------------- 20:37:53 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 20:37:53 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:53 [2KMap 1 container RUNNING 1 0 1 0 0 0 20:37:53 [2K---------------------------------------------------------------------------------------------- 20:37:53 [2K[31;1mVERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 4.74 s 20:37:53 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 [7A[2K---------------------------------------------------------------------------------------------- 20:37:54 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 20:37:54 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 [2KMap 1 .......... container SUCCEEDED 1 1 0 0 0 0 20:37:54 [2K---------------------------------------------------------------------------------------------- 20:37:54 [2K[31;1mVERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 5.75 s 20:37:54 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 [7A[2K---------------------------------------------------------------------------------------------- 20:37:54 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 20:37:54 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 [2KMap 1 .......... container SUCCEEDED 1 1 0 0 0 0 20:37:54 [2K---------------------------------------------------------------------------------------------- 20:37:54 [2K[31;1mVERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 5.88 s 20:37:54 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 INFO : Starting task [Stage-7:CONDITIONAL] in serial mode 20:37:54 INFO : Stage-4 is selected by condition resolver. 20:37:54 INFO : Stage-3 is filtered out by condition resolver. 20:37:54 INFO : Stage-5 is filtered out by condition resolver. 20:37:54 INFO : Starting task [Stage-4:MOVE] in serial mode 20:37:54 INFO : Moving data to directory hdfs://localhost:20500/test-warehouse/hive_warm_up_tbl/.hive-staging_hive_2023-12-14_20-37-40_924_4404023279633860277-1/-ext-10000 from hdfs://localhost:20500/test-warehouse/hive_warm_up_tbl/.hive-staging_hive_2023-12-14_20-37-40_924_4404023279633860277-1/-ext-10002 20:37:54 INFO : Starting task [Stage-2:DEPENDENCY_COLLECTION] in serial mode 20:37:54 INFO : Starting task [Stage-0:MOVE] in serial mode 20:37:54 INFO : Loading data to table default.hive_warm_up_tbl from hdfs://localhost:20500/test-warehouse/hive_warm_up_tbl/.hive-staging_hive_2023-12-14_20-37-40_924_4404023279633860277-1/-ext-10000 20:37:54 INFO : Completed executing command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f); Time taken: 12.35 seconds 20:37:54 INFO : OK 20:37:54 [7A[2K---------------------------------------------------------------------------------------------- 20:37:54 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED 20:37:54 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 [2KMap 1 .......... container SUCCEEDED 1 1 0 0 0 0 20:37:54 [2K---------------------------------------------------------------------------------------------- 20:37:54 [2K[31;1mVERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 5.88 s 20:37:54 [22;0m[2K---------------------------------------------------------------------------------------------- 20:37:54 1 row affected (13.325 seconds) 20:37:54 Beeline version 3.1.3000.7.2.18.0-369 by Apache Hive 20:37:54 Closing: 0: jdbc:hive2://localhost:11050/default;