Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22608

MVCC's writeEntry didn't complete and make MVCC stuck

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • in-memory-compaction
    • None

    Description

      2019-06-20,05:03:44,917 ERROR [RpcServer.default.RWQ.Fifo.write.handler=61,queue=1,port=22600] org.apache.hadoop.hbase.regionserver.HRegion: Asked to modify this region's (xxxxxx,,1560481375170.10b01c12d58ce75c9aaf1ac15cc2a7f3.) memStoreSizing to a negative value which is incorrect. Current memStoreSizing=-1686222, delta=1489930
      java.lang.Exception
      at org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1317)
      at org.apache.hadoop.hbase.regionserver.HRegion.incMemStoreSize(HRegion.java:1295)
      at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3316)
      at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore(HRegion.java:3821)
      at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4248)
      at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4179)
      at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4109)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1059)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:991)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:954)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2833)
      at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014)
      at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
      at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
      at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
      

      See https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3728

      @Override
      public WriteEntry writeMiniBatchOperationsToMemStore(
          final MiniBatchOperationInProgress<Mutation> miniBatchOp, @Nullable WriteEntry writeEntry)
          throws IOException {
        if (writeEntry == null) {
          writeEntry = region.mvcc.begin();
        }
        super.writeMiniBatchOperationsToMemStore(miniBatchOp, writeEntry.getWriteNumber());
        return writeEntry;
      }
      

      super.writeMiniBatchOperationsToMemStore throw a exception and the new writeEntry cannot be complete and make the MVCC stuck.

       

      And we meet this problem when enable in-memory compaction. But that should be another issue and need to dig more.

      Attachments

        Issue Links

          Activity

            People

              openinx Zheng Hu
              zghao Guanghao Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: