Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.5.7
-
None
-
None
Description
Issue: Data Inconsistency Between Zookeeper Leader and zookeeper Followers. zookeeper followers and zookeeper leaders have other information. We try to delete the information from the follower's, but information not present in zookeeper leader, it's throwing error like Node does not exist:
Expected behaviour: Data consistency between zookeeper leader and Zookeeper followers should be same.
Steps followed as part of troubleshooting:
We have 5 zookeepers in clusters.
Step1: verified all zookeepers are following the leader or not?. As per below information its following all 4 zookeepers to zookeeper leader
zk_version 3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/11/2020 11:30 GMT
zk_avg_latency 0
zk_max_latency 823
zk_min_latency 0
zk_packets_received 30214264
zk_packets_sent 32424272
zk_num_alive_connections 7
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 75190
zk_watch_count 21394
zk_ephemerals_count 793
zk_approximate_data_size 24706628
zk_open_file_descriptor_count 281
zk_max_file_descriptor_count 4096
zk_followers 4
zk_synced_followers 4
zk_pending_syncs 0
zk_last_proposal_size 166
zk_max_proposal_size 121947
zk_min_proposal_size 32
Step 2: Verified znode in all the zookeepers , but we are not getting same information from zookeeper leader and followers.
Step 3: Try to delete the Zookeeper node and received below error. Also, we are suspecting that when trying to delete the info of znode, it's trying to reach zookeeper leader and throwing Node does not exist error.
Error:
14:04:54.769 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
14:04:54.775 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
14:04:54.824 [main-SendThread(11.111.226.146:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 11-111-226-146.ebiz.verizon.com/11.111.226.146:2181. Will not attempt to authenticate using SASL (unknown error)
14:04:54.831 [main-SendThread(11.111.226.146:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /11.111.225.75:38804, server: 11-111-226-146.ebiz.verizon.com/11.111.20.146:2181
14:04:54.835 [main-SendThread(11.111.226.146:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 11-111-226-146.ebiz.verizon.com/11.111.226.146:2181, sessionid = 0x500001bbbeb0651, negotiated timeout = 20000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Node does not exist: /namespace/$tenant/$Namespace/$zk-path