Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2325

Data inconsistency if all snapshots empty or missing

    XMLWordPrintableJSON

Details

    Description

      When loading state from snapshots on startup, FileTxnSnapLog.java ignores the result of FileSnap.deserialize, which is -1L if no valid snapshots are found. Recovery proceeds with dt.lastProcessed == 0, its initial value.

      The result is that Zookeeper will process the transaction logs and then begin serving requests with a different state than the rest of the ensemble.

      To reproduce:
      In a healthy zookeeper cluster of size >= 3, shut down one node.
      Either delete all snapshots for this node or change all to be empty files.
      Restart the node.

      We believe this can happen organically if a node runs out of disk space.

      Attachments

        1. zk.patch
          1 kB
          Benjamin Reed
        2. ZOOKEEPER-2325.001.patch
          0.8 kB
          Andrew Grasso
        3. ZOOKEEPER-2325-test.patch
          5 kB
          Andrew Grasso

        Issue Links

          Activity

            People

              agrasso Andrew Grasso
              agrasso Andrew Grasso
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Time Spent - 10m Remaining Estimate - 50m
                  50m
                  Logged:
                  Time Spent - 10m Remaining Estimate - 50m
                  10m