Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.4.13
-
None
-
None
-
Debian, Java 8
Description
I am working on a testing tool for distributed systems. I tested Zookeeper, enforcing different possible orderings of events. I encountered some inconsistencies in the election of the leader. Here are the logs of 3 completed executions.
I am wondering if these behaviors are expected or not.
1) More than one node consider themselves leaders:
NodeCrashEvent{id=1, nodeId=0}
NodeStartEvent{id=7, nodeId=0}
MessageEvent{id=8, predecessors=[7], from=0, to=0, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=9, predecessors=[8, 7], from=0, to=1, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=10, predecessors=[9, 7], from=0, to=2, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=5, predecessors=[], from=1, to=0, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=12, predecessors=[5, 10, 7], from=0, to=0, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=13, predecessors=[12, 5, 7], from=0, to=1, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=14, predecessors=[5, 13, 7], from=0, to=2, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=11, predecessors=[5], from=1, to=1, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=15, predecessors=[11], from=1, to=2, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=6, predecessors=[], from=2, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
Node 1 state: LEADING
Node 1 final vote: Vote{leader=1, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=17, predecessors=[6, 14, 7], from=0, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=18, predecessors=[17, 6, 7], from=0, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=19, predecessors=[18, 6, 7], from=0, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=20, predecessors=[18], from=1, to=0, leader=1, state=LEADING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=16, predecessors=[6], from=2, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=22, predecessors=[16, 20], from=1, to=2, leader=1, state=LEADING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=21, predecessors=[16], from=2, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
Node 0 state: FOLLOWING
Node 0 final vote: Vote{leader=2, zxid=0, electionEpoch=1, peerEpoch=0}
Node 2 state: LEADING
Node 2 final vote: Vote{leader=2, zxid=0, electionEpoch=1, peerEpoch=0}
2) There are some nodes that follow nodes other than the leaders:
NodeCrashEvent{id=1, nodeId=0}
NodeStartEvent{id=7, nodeId=0}
MessageEvent{id=8, predecessors=[7], from=0, to=0, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=9, predecessors=[8, 7], from=0, to=1, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=10, predecessors=[9, 7], from=0, to=2, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=5, predecessors=[], from=1, to=0, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=12, predecessors=[5, 10, 7], from=0, to=0, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=13, predecessors=[12, 5, 7], from=0, to=1, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=14, predecessors=[5, 13, 7], from=0, to=2, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
Node 0 state: FOLLOWING
Node 0 final vote: Vote{leader=1, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=11, predecessors=[5], from=1, to=1, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=15, predecessors=[11], from=1, to=2, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=6, predecessors=[], from=2, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=17, predecessors=[6, 7], from=0, to=2, leader=1, state=FOLLOWING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=16, predecessors=[6], from=2, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=19, predecessors=[16, 15], from=1, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=20, predecessors=[16, 19], from=1, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=22, predecessors=[16, 20], from=1, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=21, predecessors=[17, 19, 7], from=0, to=1, leader=1, state=FOLLOWING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=18, predecessors=[16], from=2, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
Node 1 state: FOLLOWING
Node 1 final vote: Vote{leader=2, zxid=0, electionEpoch=1, peerEpoch=0}
Node 2 state: LEADING
Node 2 final vote: Vote{leader=2, zxid=0, electionEpoch=1, peerEpoch=0}
3) There are some nodes that neither following nor leading
NodeCrashEvent{id=3, nodeId=2}
NodeStartEvent{id=7, nodeId=2}
MessageEvent{id=8, predecessors=[7], from=2, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=9, predecessors=[8, 7], from=2, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=10, predecessors=[9, 7], from=2, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=5, predecessors=[], from=1, to=0, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=11, predecessors=[5], from=1, to=1, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=12, predecessors=[11], from=1, to=2, leader=1, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=13, predecessors=[12, 9], from=1, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=14, predecessors=[9, 13], from=1, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=15, predecessors=[9, 14], from=1, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=4, predecessors=[], from=0, to=0, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=16, predecessors=[4], from=0, to=1, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=17, predecessors=[16], from=0, to=2, leader=0, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=18, predecessors=[8, 17], from=0, to=0, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=19, predecessors=[8, 18], from=0, to=1, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
Node 2 state: LEADING
Node 2 final vote: Vote{leader=2, zxid=0, electionEpoch=1, peerEpoch=0}
Node 1 state: FOLLOWING
Node 1 final vote: Vote{leader=2, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=20, predecessors=[8, 19], from=0, to=2, leader=2, state=LOOKING, zxid=0, electionEpoch=1, peerEpoch=0}
MessageEvent{id=21, predecessors=[20, 7], from=2, to=0, leader=2, state=LEADING, zxid=0, electionEpoch=1, peerEpoch=1}