Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
None
-
None
-
None
Description
2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930 java.lang.Exception at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317) at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
@Override public WriteEntry writeMiniBatchOperationsToMemStore( final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry) throws IOException { if (writeEntry == null) { writeEntry = region.mvcc.begin(); } super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber()); return writeEntry; }
super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.
And we meet this problem when enable in-memory compaction. But that should be another issue and need to dig more.
Attachments
Issue Links
- is related to
-
HBASE-23084 Out-of-order adjacent keys causes regionserver crash
- Open
-
HBASE-22995 The TestHRegionWithInMemoryFlush is broken
- Resolved