Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
1.7.0
-
None
-
None
Description
When concurrent assign a region, the RS may abort when reporting region state to master encounters transit region state error,
2021-11-26 04:40:54,689 INFO [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65] regionserver.HRegionServer: Post open deploy tasks for growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65. 2021-11-26 04:40:54,689 INFO [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65] regionserver.HRegionServer: Failed to transition {ENCODED => ee1df69da9c7fb7698d2972f81d40b65, NAME => 'growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65.', STARTKEY => 'cc', ENDKEY => 'cd'} to OPENED: ee1df69da9c7fb7698d2972f81d40b65 is not pending open on zf-data-hbase250.mt,16020,1637872783437 2021-11-26 04:40:54,689 FATAL [PostOpenDeployTasks:ee1df69da9c7fb7698d2972f81d40b65] regionserver.HRegionServer: ABORTING region server zf-data-hbase250.mt,16020,1637872783437: Exception running postOpenDeployTasks; region=ee1df69da9c7fb7698d2972f81d40b65 java.io.IOException: Failed to report opened region to master: growthosudx:growthos-udx_dmx_supply_waimaisku_application_basic,cc,1631020952330.ee1df69da9c7fb7698d2972f81d40b65. at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2387) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:327)
In the master log, we can see that the region is being assigned by both the bulk assigner and the SCP.
This steps can reproduce the problem:
- thread 1 calls Assignment#assign to assign region A, and calls RegionStates#updateRegionsState and stops before the synchronize;
- thread 2 calls RegionStates#serverOffline and gets the lock, it sees that region A is on the processed dead server and add region A to the regionsToOffline list(region A is not in the RIT cache now);
- thread 2 prepare to execute RegionStates#regionOffline, and released the lock;
- thread 1 get the lock, and add region A to the RIT cache, and also update the region state by RegionStates#putRegionState, set the region state from OFFLINE to PENDING_OPEN, then release the lock;
- thread2 execute RegionStates#regionOffline, acquires the lock and sets the region state from PENDING_OPEN to OFFLINE;
- the RS that region A assigned by thread 1 report region A should be transited to OPENED, but finds that region A last state is OFFLINE, error occures, the RS abort itself.
Attachments
Attachments
Issue Links
- links to