Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21378

[hbck2] add --skip version check to hbck2 tool (checkHBCKSupport blocks assigning hbase:meta or hbase:namespace when master is not initialized)

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      Adds a general -s,--skip option to hbck2 so you can run the command it first checking it is version compatible.

      Should not be needed going forward but in the spirt of our not knowing all the conditions under which we'll be trying to run hbck2, adding it.
      Show
      Adds a general -s,--skip option to hbck2 so you can run the command it first checking it is version compatible. Should not be needed going forward but in the spirt of our not knowing all the conditions under which we'll be trying to run hbck2, adding it.

      Description

      When I encounter the scenario that hbase:namespace is not online.

      2018-10-24,14:38:16,910 WARN org.apache.hadoop.hbase.master.HMaster: hbase:namespace,,1529933109115.7e0801c8232b2dc15face54532056076. is NOT online; state={7e0801c8232b2dc15face54532056076 state=OPEN, ts=1540363033384, server=c4-hadoop-tst-st30.bj,29100,1540348649479}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
      

      Then I tried to assign it manually, but it throws PleaseHoldException.

      Wed Oct 24 15:26:52 CST 2018, RpcRetryingCaller{globalStartTime=1540365754487, pause=200, maxAttempts=16}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
      	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3064)
      	at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:934)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      
      
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:144)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3133)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3125)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.getClusterMetrics(HBaseAdmin.java:2161)
      	at org.apache.hbase.HBCK2.checkHBCKSupport(HBCK2.java:98)
      	at org.apache.hbase.HBCK2.run(HBCK2.java:364)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      	at org.apache.hbase.HBCK2.main(HBCK2.java:447)
      Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
      	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3064)
      	at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:934)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
      	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
      	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361)
      	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:349)
      	at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101)
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
      

      Then I check the code and found it is because of checkHBCKSupport(), I assign hbase:namespace successfully by skipping this check. Thus I think the tool need an option to skip this check.

        Attachments

          Activity

            People

            • Assignee:
              tianjingyun Jingyun Tian
              Reporter:
              tianjingyun Jingyun Tian
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: