Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22017

Master Fails to become active due to the data race bug in region server

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None

    Description

      Test cluster: hadoop11(master), hadoop14(slave), haoop15(slave).

      before code execute at org.apache.hadoop.hbase.regionserver.HStore#getScanner(function)#2027(line number), hadoop15 shutdown, then master startup fails

      2019-03-06 01:36:17,040 ERROR [master/hadoop11:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master hadoop11,16000,1551807353275: Unhandled exception. Starting shutdown. *****
      org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '3449673378019934209' does not exist
      at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:224)
      at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3434)
      at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
      at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
      at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
      at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361)
      at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:349)
      at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:344)
      at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
      at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
      at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
      at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
      at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
      at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
      at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      
      

      Attachments

        1. logs.zip
          30 kB
          lujie
        2. fixedlogs.zip
          66 kB
          lujie
        3. 0002-fix-HBASE-22017.patch
          2 kB
          lujie
        4. 0001-fix-HBASE-22017.patch
          2 kB
          lujie

        Issue Links

          Activity

            People

              Unassigned Unassigned
              xiaoheipangzi lujie
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: