Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.3.0
-
None
-
None
Description
When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to start due to a known zookeeper failure - ZOOKEEPER-3056.
The error message is shown below:
2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master exitingjava.io.IOException: No snapshot found, but there are log entries. Something is broken! at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240) at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290) at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450) at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764) at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259) at org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
It can be reproduced through the following steps:
- Start a single-node HBase 2.2 with the following configuration.
<property> <name>hbase.rootdir</name> <value>/var/lib/hbase</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/var/lib/zookeeper</value> </property>
- Use HBase performance evaluation tool to generate data.
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1
- Upgrade the node to 2.3 with the same configuration. The new version node failed to start because of the zookeeper.
HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using zookeeper-3.5.7.jar.
The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we upgrade the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade the configuration file of zookeeper-3.5.7.jar?
Attachments
Issue Links
- is broken by
-
HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7
-
- Resolved
-