Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-27908

Can't get connection to ZooKeeper

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 1.4.13
    • None
    • build
    • None

    Description

      I am using Hbase cluster along with apache kylin, the connection between Edge node and the Hbase cluster is good.
      following are the logs from Kylin side which shows Error exception:

      java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1275361: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase 
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
          at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4551)
          at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:561)
          at org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:585)
          at org.apache.kylin.storage.hbase.steps.HFileOutputFormat3.configureIncrementalLoad(HFileOutputFormat3.java:328)
          at org.apache.kylin.storage.hbase.steps.CubeHFileJob.run(CubeHFileJob.java:101)
          at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:144)
          at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
          at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
          at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
          at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:750)
      Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1618)
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1638)
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1795)
          at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:140)
          ... 13 more
      Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:971)
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:566)
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1567)
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1609)
          ... 17 more
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
          at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
          at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
          at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
          at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:220)
          at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:425)
          at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:960)
          ... 20 more 

      Following are the logs from Hbase cluster master NOde which accepts the connection from Edge NOde(Kylin):

      2023-06-05 10:00:30,336 [myid:0] - INFO [CommitProcessor:0:NIOServerCnxn@1056] - Closed socket connection for client /10.127.2.201:37328 which had sessionid 0x7311c000c
      2023-06-05 13:14:48,346 [myid:0] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
      2023-06-05 13:14:48,346 [myid:0] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
      2023-06-05 13:22:39,872 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /10.127.2.233:42364
      2023-06-05 13:22:39,873 [myid:0] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@949] - Client attempting to establish new session at /10.127.2.233:42364
      2023-06-05 13:22:39,874 [myid:0] - INFO [CommitProcessor:0:ZooKeeperServer@694] - Established session 0x7311c0022 with negotiated timeout 40000 for client /10.127.2.233:42364 

      have check all the permission in HDFS and S3. Any leads would be really appreciated.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ibrar1761 Ibrar Ahmed
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: