Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2247

Zookeeper service becomes unavailable when leader fails to write transaction log

    XMLWordPrintableJSON

Details

    Description

      Zookeeper service becomes unavailable when leader fails to write transaction log. Bellow are the exceptions

      2015-08-14 15:41:18,556 [myid:100] - ERROR [SyncThread:100:ZooKeeperCriticalThread@48] - Severe unrecoverable error, from thread : SyncThread:100
      java.io.IOException: Input/output error
      	at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
      	at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
      	at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:376)
      	at org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:331)
      	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:380)
      	at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:563)
      	at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:178)
      	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
      2015-08-14 15:41:18,559 [myid:100] - INFO  [SyncThread:100:ZooKeeperServer$ZooKeeperServerListenerImpl@500] - Thread SyncThread:100 exits, error code 1
      2015-08-14 15:41:18,559 [myid:100] - INFO  [SyncThread:100:ZooKeeperServer@523] - shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:SessionTrackerImpl@232] - Shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:LeaderRequestProcessor@77] - Shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:PrepRequestProcessor@1035] - Shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:ProposalRequestProcessor@88] - Shutting down
      2015-08-14 15:41:18,561 [myid:100] - INFO  [SyncThread:100:CommitProcessor@356] - Shutting down
      2015-08-14 15:41:18,561 [myid:100] - INFO  [CommitProcessor:100:CommitProcessor@191] - CommitProcessor exited loop!
      2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:Leader$ToBeAppliedRequestProcessor@915] - Shutting down
      2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:FinalRequestProcessor@646] - shutdown of request processor complete
      2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:SyncRequestProcessor@191] - Shutting down
      2015-08-14 15:41:18,563 [myid:100] - INFO  [ProcessThread(sid:100 cport:-1)::PrepRequestProcessor@159] - PrepRequestProcessor exited loop!
      

      After this exception Leader server still remains leader. After this non recoverable exception the leader should go down and let other followers become leader.

      Attachments

        1. ZOOKEEPER-2247-01.patch
          37 kB
          Mohammad Arshad
        2. ZOOKEEPER-2247-02.patch
          37 kB
          Mohammad Arshad
        3. ZOOKEEPER-2247-03.patch
          45 kB
          Mohammad Arshad
        4. ZOOKEEPER-2247-04.patch
          45 kB
          Mohammad Arshad
        5. ZOOKEEPER-2247-05.patch
          35 kB
          Mohammad Arshad
        6. ZOOKEEPER-2247-06.patch
          19 kB
          Mohammad Arshad
        7. ZOOKEEPER-2247-07.patch
          21 kB
          Rakesh Radhakrishnan
        8. ZOOKEEPER-2247-09.patch
          21 kB
          Rakesh Radhakrishnan
        9. ZOOKEEPER-2247-10.patch
          21 kB
          Rakesh Radhakrishnan
        10. ZOOKEEPER-2247-11.patch
          21 kB
          Rakesh Radhakrishnan
        11. ZOOKEEPER-2247-12.patch
          22 kB
          Rakesh Radhakrishnan
        12. ZOOKEEPER-2247-13.patch
          24 kB
          Rakesh Radhakrishnan
        13. ZOOKEEPER-2247-14.patch
          25 kB
          Rakesh Radhakrishnan
        14. ZOOKEEPER-2247-15.patch
          24 kB
          Rakesh Radhakrishnan
        15. ZOOKEEPER-2247-16.patch
          30 kB
          Rakesh Radhakrishnan
        16. ZOOKEEPER-2247-17.patch
          28 kB
          Rakesh Radhakrishnan
        17. ZOOKEEPER-2247-18.patch
          28 kB
          Rakesh Radhakrishnan
        18. ZOOKEEPER-2247-19.patch
          28 kB
          Rakesh Radhakrishnan
        19. ZOOKEEPER-2247-20.patch
          28 kB
          Rakesh Radhakrishnan
        20. ZOOKEEPER-2247-21.patch
          29 kB
          Rakesh Radhakrishnan
        21. ZOOKEEPER-2247-22.patch
          29 kB
          Rakesh Radhakrishnan
        22. ZOOKEEPER-2247-23.patch
          29 kB
          Rakesh Radhakrishnan
        23. ZOOKEEPER-2247-b3.5.patch
          12 kB
          Flavio Paiva Junqueira
        24. ZOOKEEPER-2247-br-3.4.patch
          30 kB
          Rakesh Radhakrishnan
        25. ZOOKEEPER-2247-br-3.4.patch
          24 kB
          Rakesh Radhakrishnan

        Issue Links

          Activity

            People

              rakeshr Rakesh Radhakrishnan
              arshad.mohammad Mohammad Arshad
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m