Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22477

Throwing exception when meta region is not in OPEN state in client registry may crash a master

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      2019-05-26 17:10:31,195 ERROR [master/asf906:0:becomeActiveMaster] helpers.MarkerIgnoringBase(159): Failed to become active master
      org.apache.hadoop.hbase.client.RetriesExhaustedException: Cannot get the location for replica0 of region for  in hbase:meta
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:335)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
      	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:263)
      	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:405)
      	at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:285)
      	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:564)
      	at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:766)
      	at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
      	at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690)
      	at org.apache.hadoop.hbase.MetaTableAccessor.fullScanRegions(MetaTableAccessor.java:220)
      	at org.apache.hadoop.hbase.master.assignment.RegionStateStore.visitMeta(RegionStateStore.java:77)
      	at org.apache.hadoop.hbase.master.assignment.AssignmentManager.loadMeta(AssignmentManager.java:1294)
      	at org.apache.hadoop.hbase.master.assignment.AssignmentManager.joinCluster(AssignmentManager.java:1255)
      	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1100)
      	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2375)
      	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:605)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Meta region is in state OPENING
      	at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$1(ZKAsyncRegistry.java:162)
      	at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:70)
      	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
      	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
      	at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:81)
      	at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:70)
      	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
      	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
      	at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
      	at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
      	... 1 more
      

      I think the logic is introduced at the time that we do not change the state of meta region on zk. But now, we will change the state of meta region, so maybe we should remove the logic.

      Attachments

        1. HBASE-22477-test.patch
          1 kB
          Duo Zhang

        Issue Links

          Activity

            People

              zhangduo Duo Zhang
              zhangduo Duo Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: