Details
Description
When an abort is triggered within the RPC call path by HRegion.RegionScannerImpl, AccessController is incorrectly apply the RPC caller identity in the RegionServerObserver.preStopRegionServer() hook. This leaves the regionserver in a non-responsive state, where its regions are not reassigned and it returns exceptions for all requests.
When an abort is triggered on the server side, we should not allow a coprocessor to reject the abort at all.
Here is a sample stack trace:
17/05/25 06:10:29 FATAL regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.security.token.TokenProvider] 17/05/25 06:10:29 WARN regionserver.HRegionServer: The region server did not stop org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'rpcuser' (global, action=ADMIN) at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:548) at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522) at org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501) at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:86) at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execShutdown(RegionServerCoprocessorHost.java:300) at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:82) at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2118) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2125) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.abortRegionServer(HRegion.java:6326) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleFileNotFound(HRegion.java:6319) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5941) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6084) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5858) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2649) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2320) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
I haven't yet evaluated which other release branches this might apply to.
I have a patch currently in progress, which I will post as soon as I complete a test case.