Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Bug
-
Impala 1.2.4
-
None
-
None
-
CDH 4.6 from Parcels, Impala 1.2.4 from Parcels
CM 4.8.1
Description
Hi, we started to get strange bug with statestore and impalad
Here is what we do.
1. connect to local impalad
2. show tables - OK
3. select from table - FAIL, ERROR: No backends configured
devops@prod-node012 ~]$ impala-shell Starting Impala Shell without Kerberos authentication Connected to prod-node012.kyc.lol.ru:21000 Server version: impalad version 1.2.4 RELEASE (build ac29ae09d66c1244fe2ceb293083723226e66c1a) Welcome to the Impala shell. Press TAB twice to see a list of available commands. Copyright (c) 2012 Cloudera, Inc. All rights reserved. (Shell build version: Impala Shell v1.2.4 (ac29ae0) built on Wed Mar 5 07:05:40 PST 2014) [prod-node012.kyc.lol.ru:21000] > connect localhost; Connected to localhost:21000 Server version: impalad version 1.2.4 RELEASE (build ac29ae09d66c1244fe2ceb293083723226e66c1a) [localhost:21000] > show tables; Query: show tables +------------------------------------------------------------------+ | name | +------------------------------------------------------------------+ | my_fav_table | | many_more_goes_here | +------------------------------------------------------------------+ Returned 121 row(s) in 0.02s [localhost:21000] > select count(*) from my_fav_table; Query: select count(*) from my_fav_table ERROR: No backends configured
Here is node012 impalad log:
15:40:49.197 INFO statestore-subscriber.cc:216 prod-node0102.kyc.lol.ru:22000: Connection with statestore lost, entering recovery mode 15:40:49.197 INFO statestore-subscriber.cc:220 Trying to re-register with statestore, attempt: 1 15:42:25.235 INFO statestore-subscriber.cc:159 Subscriber registration ID: 7f470575215239dc:63643be6ab1fb98c 15:42:25.235 INFO statestore-subscriber.cc:224 Reconnected to statestore. Exiting recovery mode 15:42:55.236 INFO statestore-subscriber.cc:216 prod-node0102.kyc.lol.ru:22000: Connection with statestore lost, entering recovery mode 15:42:55.236 INFO statestore-subscriber.cc:220 Trying to re-register with statestore, attempt: 1
Here is statestore log:
15:40:19.110 INFO statestore.cc:329 Subscriber 'prod-node012.kyc.lol.ru:22000' registered (registration id: 98437526ee48f64f:7fcb9fc4796414af) 15:40:49.111 INFO statestore.cc:298 Registering: prod-node012.kyc.lol.ru:22000 15:42:25.148 INFO statestore.cc:329 Subscriber 'prod-node012.kyc.lol.ru:22000' registered (registration id: f41c966d3733c1c:655a2ccea68edebb)
What it could be?
We have more roles (TT, DN) running on prod-node012 and there are no problems with them. We did upgrade to Impala 1.2.4, CDH 4.6 a month ago and problem appeared today. I have no idea what to do. how can we debug it?