Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Duplicate
-
None
-
None
-
None
-
ghx-label-4
Description
Error Message assert 'Initial admission queue reason: Waiting for executors to start' in 'Query (id=d9484fef8ecacec7:5278c53400000000):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...0)\n - NumRowsFetchedFromCache: 0 (0)\n - RowMaterializationRate: 0\n - RowMaterializationTimer: 0.000ns\n' Stacktrace custom_cluster/test_executor_groups.py:315: in test_sequential_startup_wait assert "Initial admission queue reason: Waiting for executors to start" in profile E assert 'Initial admission queue reason: Waiting for executors to start' in 'Query (id=d9484fef8ecacec7:5278c53400000000):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...0)\n - NumRowsFetchedFromCache: 0 (0)\n - RowMaterializationRate: 0\n - RowMaterializationTimer: 0.000ns\n' Standard Error -- 2019-10-20 01:57:00,039 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=1 --log_dir=/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests --log_level=1 --use_exclusive_coordinators '--impalad_args= -executor_groups=coordinator ' --impalad_args=--default_query_options= 01:57:01 MainThread: Starting impala cluster without executors 01:57:01 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) 01:57:01 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/statestored.INFO 01:57:03 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/catalogd.INFO 01:57:04 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/impalad.INFO 01:57:08 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 01:57:08 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 01:57:08 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:08 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused')) 01:57:10 MainThread: Debug webpage did not become available in expected time. 01:57:10 MainThread: Waiting for num_known_live_backends=1. Current value: None 01:57:11 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 01:57:11 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:11 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused')) 01:57:13 MainThread: Debug webpage did not become available in expected time. 01:57:13 MainThread: Waiting for num_known_live_backends=1. Current value: None 01:57:14 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 01:57:14 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:14 MainThread: num_known_live_backends has reached value: 1 01:57:15 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 0 executors). -- 2019-10-20 01:57:15,401 DEBUG MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) -- 2019-10-20 01:57:15,402 INFO MainThread: Getting metric: statestore.live-backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25010 -- 2019-10-20 01:57:15,409 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:15,421 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 2 -- 2019-10-20 01:57:15,421 DEBUG MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 -- 2019-10-20 01:57:15,423 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:15,427 INFO MainThread: num_known_live_backends has reached value: 1 SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_sequential_startup_wait; -- connecting to: localhost:21000 -- connecting to localhost:21050 with impyla -- 2019-10-20 01:57:15,786 INFO MainThread: Closing active operation -- 2019-10-20 01:57:15,793 INFO MainThread: Adding 1 executors to group default-pool-group1 with minimum size 3 -- 2019-10-20 01:57:15,794 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=0 --log_dir=/data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests --log_level=1 --add_executors --impalad_args=-admission_control_slots=0 --impalad_args=-executor_groups=default-pool-group1:3 --impalad_args=--default_query_options= 01:57:17 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 01:57:17 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6-shard1/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO 01:57:20 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:20 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:20 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:20 MainThread: Waiting for num_known_live_backends=2. Current value: 1 01:57:21 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:21 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:21 MainThread: Waiting for num_known_live_backends=2. Current value: 1 01:57:23 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:23 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:23 MainThread: Waiting for num_known_live_backends=2. Current value: 1 01:57:24 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:24 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:24 MainThread: Waiting for num_known_live_backends=2. Current value: 1 01:57:25 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:25 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 01:57:25 MainThread: num_known_live_backends has reached value: 2 01:57:25 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:57:25 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25001 01:57:25 MainThread: num_known_live_backends has reached value: 2 01:57:25 MainThread: Impala Cluster Running with 1 nodes (0 coordinators, 1 executors). -- 2019-10-20 01:57:26,163 DEBUG MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) -- 2019-10-20 01:57:26,164 INFO MainThread: Getting metric: statestore.live-backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25010 -- 2019-10-20 01:57:26,170 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:26,181 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 3 -- 2019-10-20 01:57:26,181 DEBUG MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 -- 2019-10-20 01:57:26,183 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:26,189 INFO MainThread: num_known_live_backends has reached value: 2 -- 2019-10-20 01:57:26,189 DEBUG MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25001 -- 2019-10-20 01:57:26,191 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:26,196 INFO MainThread: num_known_live_backends has reached value: 2 -- 2019-10-20 01:57:26,197 INFO MainThread: Getting metric: cluster-membership.backends.total from shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com:25000 -- 2019-10-20 01:57:26,199 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:26,214 INFO MainThread: Metric 'cluster-membership.backends.total' has reached desired value: 2 -- 2019-10-20 01:57:26,216 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- 2019-10-20 01:57:26,230 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-002d.vpc.cloudera.com -- executing async: localhost:21000 select sleep(4); -- 2019-10-20 01:57:26,663 INFO MainThread: Started query d9484fef8ecacec7:5278c53400000000 -- getting runtime profile operation: <tests.common.impala_connection.OperationHandle object at 0x5164dd0>
Attachments
Issue Links
- duplicates
-
IMPALA-9202 Fix flakiness in TestExecutorGroups
- Resolved