Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20146

Regions are stuck while opening when WAL is disabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.3.2, 1.4.3, 2.0.0, 1.2.7
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      On a running cluster we had set hbase.regionserver.hlog.enabled to false, to disable the WAL for complete cluster, after restarting HBase service, regions are not getting opened leading to HMaster abort as Namespace table regions are not getting assigned. 

      jstack for region open:

      "RS_OPEN_PRIORITY_REGION-BLR1000006595:16045-1" #159 prio=5 os_prio=0 tid=0x00007fdfa4341000 nid=0x419d waiting on condition [0x00007fdfa0467000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x0000000087554448> (a java.util.concurrent.CountDownLatch$Sync)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
      at org.apache.hadoop.hbase.wal.WALKey.getWriteEntry(WALKey.java:98)
      at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:131)
      at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:88)
      at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:1026)
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6849)
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6803)
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6774)
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6730)
      at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6681)
      at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
      at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
      at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      

      This used to work with HBase 1.0.2 version.

        Attachments

        1. HBASE-20146.patch
          5 kB
          Ashish Singhi
        2. HBASE-20146.v1.patch
          5 kB
          Ashish Singhi
        3. HBASE-20146-addendum.patch
          2 kB
          Ashish Singhi

          Issue Links

            Activity

              People

              • Assignee:
                ashish singhi Ashish Singhi
                Reporter:
                ashish singhi Ashish Singhi
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: