Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21932

Use Runtime.getRuntime().halt to terminate regionserver when abort timeout

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 1.5.0, 2.2.0, 2.0.4, 2.1.3, 2.3.0
    • Fix Version/s: 3.0.0, 1.5.0, 2.2.0, 2.0.5, 2.3.0, 2.1.4
    • Component/s: None
    • Labels:
      None

      Description

      Find one case when run ITBLL. And the regionserver hang when abort as need to waitAllRegionClose. But System.exit need to run shutdown hook. And the hook can't finished as the RS thread is still alive. So use Runtime.getRuntime().halt to terminate regionserver and this method will not run any shutdown hook.

       

      Stack trace of RegionServer

      "regionserver/c4-hadoop-tst-st27:29100" #29 daemon prio=5 os_prio=0 tid=0x00007f03d6066670 nid=0x1afe7 waiting on condition [0x00007eff7ce4e000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1453)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1439)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1133)
      at java.lang.Thread.run(Thread.java:745)
      
      "Thread-10" #68 prio=5 os_prio=0 tid=0x00007f02bc7bbeb0 nid=0x141d3 in Object.wait() [0x00007eff486e4000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Thread.join(Thread.java:1249)
      - locked <0x00000005c02da008> (a java.lang.Thread)
      at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:113)
      at org.apache.hadoop.hbase.util.Threads.shutdown(Threads.java:101)
      at org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread.run(ShutdownHook.java:116)
      at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
      
      "Abort regionserver monitor" #26361 daemon prio=5 os_prio=0 tid=0x00007f00605a43d0 nid=0x1111e in Object.wait() [0x00007eff48a6a000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Thread.join(Thread.java:1249)
      - locked <0x00000005c0b55c80> (a org.apache.hadoop.util.ShutdownHookManager$1)
      at java.lang.Thread.join(Thread.java:1323)
      at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
      at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
      at java.lang.Shutdown.runHooks(Shutdown.java:123)
      at java.lang.Shutdown.sequence(Shutdown.java:167)
      at java.lang.Shutdown.exit(Shutdown.java:212)
      - locked <0x00000005c00451b0> (a java.lang.Class for java.lang.Shutdown)
      at java.lang.Runtime.exit(Runtime.java:109)
      at java.lang.System.exit(System.java:971)
      at org.apache.hadoop.hbase.regionserver.HRegionServer$SystemExitWhenAbortTimeout.run(HRegionServer.java:3792)
      at java.util.TimerThread.mainLoop(Timer.java:555)
      at java.util.TimerThread.run(Timer.java:505)
      
      "main" #1 prio=5 os_prio=0 tid=0x00007f03d40110f0 nid=0x1ac78 in Object.wait() [0x00007f03d813a000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000005c02da008> (a java.lang.Thread)
      at java.lang.Thread.join(Thread.java:1249)
      - locked <0x00000005c02da008> (a java.lang.Thread)
      at java.lang.Thread.join(Thread.java:1323)
      at org.apache.hadoop.hbase.util.HasThread.join(HasThread.java:92)
      at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:65)
      at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3044)
      

        Attachments

        1. HBASE-21932.branch-2.1.001.patch
          1 kB
          Guanghao Zhang
        2. HBASE-21932.master.001.patch
          1 kB
          Guanghao Zhang
        3. HBASE-21932.master.002.patch
          1 kB
          Guanghao Zhang

          Issue Links

            Activity

              People

              • Assignee:
                zghao Guanghao Zhang
                Reporter:
                zghao Guanghao Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: