Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19694

The initialization order for a fresh cluster is incorrect

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-beta-1, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The cluster id will set once we become the active master in finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and finishActiveMasterInitialization are both called in a thread to make the constructor of HMaster return without blocking. And since HMaster itself is also a HRegionServer, it will create a Connection and then start calling reportForDuty. And when creating the ConnectionImplementation, we will read the cluster id from zk, but the cluster id may have not been set yet since it is set in another thread, we will get an exception and use the default cluster id instead.

      I always get this when running UTs which will start a mini cluster

      2018-01-03 15:16:37,916 WARN  [M:0;zhangduo-ubuntu:32848] client.ConnectionImplementation(528): Retrieve cluster id failed
      java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid
      	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
      	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
      	at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
      	at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:286)
      	at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.<init>(ConnectionUtils.java:141)
      	at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.<init>(ConnectionUtils.java:137)
      	at org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938)
      	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      	at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163)
      	at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311)
      	... 1 more
      

        Attachments

        1. 0001-HBASE-19694.addendum.branch-2.patch
          2 kB
          Michael Stack
        2. HBASE-19694.branch-2.012.patch
          13 kB
          Michael Stack
        3. HBASE-19694.branch-2.011.patch
          13 kB
          Michael Stack
        4. HBASE-19694.branch-2.010.patch
          33 kB
          Michael Stack
        5. HBASE-19694.branch-2.009.patch
          28 kB
          Michael Stack
        6. HBASE-19694.branch-2.008.patch
          24 kB
          Michael Stack
        7. HBASE-19694.branch-2.007.patch
          24 kB
          Michael Stack
        8. HBASE-19694.branch-2.006.patch
          9 kB
          Michael Stack
        9. HBASE-19694.branch-2.005.patch
          13 kB
          Michael Stack
        10. HBASE-19694.branch-2.004.patch
          12 kB
          Michael Stack
        11. HBASE-19694.branch-2.003.patch
          12 kB
          Michael Stack
        12. HBASE-19694.branch-2.002.patch
          12 kB
          Michael Stack
        13. HBASE-19694.branch-2.001.patch
          11 kB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                zhangduo Duo Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: