Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20146

Regions are stuck while opening when WAL is disabled

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.3.1
    • 1.3.2, 1.4.3, 2.0.0, 1.2.7
    • wal
    • None
    • Reviewed

    Description

      On a running cluster we had set hbase.regionserver.hlog.enabled to false, to disable the WAL for complete cluster, after restarting HBase service, regions are not getting opened leading to HMaster abort as Namespace table regions are not getting assigned. 

      jstack for region open:

      "RS_OPEN_PRIORITY_REGION-BLR1000006595:16045-1" #159 prio=5 os_prio=0 tid=0x00007fdfa4341000 nid=0x419d waiting on condition [0x00007fdfa0467000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000087554448> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.hadoop.hbase.wal.WALKey.getWriteEntry(WALKey.java:98)
at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:131)
at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:88)
at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionOpenMarker(HRegion.java:1026)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6849)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6803)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6774)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6730)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6681)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

      This used to work with HBase 1.0.2 version.

      Attachments

        1. HBASE-20146.patch
          5 kB
          Ashish Singhi
        2. HBASE-20146.v1.patch
          5 kB
          Ashish Singhi
        3. HBASE-20146-addendum.patch
          2 kB
          Ashish Singhi

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue HBASE-14465 Backport 'Allow rowlock to be reader/write' to branch-1

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Task - A task that needs to be done. HBASE-19404 Fix DisabledWALProvider

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              ashish singhi Ashish Singhi
              ashish singhi Ashish Singhi
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: