[HBASE-27251] Rolling back from 2.5.0-SNAPSHOT to 2.4.13 fails due to `File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo` - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.5.0
Fix Version/s: 2.4.14
Component/s: master
Labels:
None

Release Note:

Hide
Before upgrading HBase cluster to 2.5.0, it is strongly recommended to upgrade the cluster to HBase-2.4.14 or later first. This is to guarantee a smooth rollback in case. Otherwise, it may run into ~~HBASE-27251~~.

Show
Before upgrading HBase cluster to 2.5.0, it is strongly recommended to upgrade the cluster to HBase-2.4.14 or later first. This is to guarantee a smooth rollback in case. Otherwise, it may run into HBASE-27251 .

Description

I was doing some perf testing with builds of 2.5.0. I rolled back to 2.4.13 and the master won't start. Stack trace ends in,

java.io.FileNotFoundException: File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo                                                                                          
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)                                                                                                                            at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)                                                                                         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2089)                                                                                                  
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:762)                                                                                         
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:458)                                        
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)                                                      at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)                                                                                           
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)                                                                                           
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)                               
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)                                                                                                                                           
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)                                                                                                                                     
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971) 
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)                                                                                                               
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)                                                                                                                                   
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)                                                                                                           
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)

When I examine the on-disk file system, I see,

nonroot@namenode-0:~$ hdfs dfs -ls /hbase/MasterData/data/master/store/
Found 3 items
drwxr-xr-x   - nonroot supergroup          0 2022-07-19 17:37 /hbase/MasterData/data/master/store/.initialized
drwxr-xr-x   - nonroot supergroup          0 2022-07-19 17:37 /hbase/MasterData/data/master/store/.tabledesc
drwxr-xr-x   - nonroot supergroup          0 2022-07-27 16:25 /hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682
nonroot@namenode-0:~$ hdfs dfs -ls /hbase/MasterData/data/master/store/.initialized/
nonroot@namenode-0:~$

Attachments

Issue Links

relates to

HBASE-26640 Reimplement master local region initialization to better work with SFT

Resolved

links to

GitHub Pull Request #4663

GitHub Pull Request #4665

Rolling back from 2.5.0-SNAPSHOT to 2.4.13 fails due to `File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo`

Details

Description

Attachments

Issue Links

Activity

People

Dates