Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15924

Enhance hbase services autorestart capability to hbase-daemon.sh

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.98.19
    • 1.4.0, 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      Now one can start hbase services with enabled "autostart/autorestart" feature in controlled fashion with the help of "--autostart-window-size" to define the window period and the "--autostart-window-retry-limit" to define the number of times the hbase services have to be restarted upon being killed/terminated abnormally within the provided window perioid.

      The following cases are supported with "autostart/autorestart":

      a) --autostart-window-size=0 and --autostart-window-retry-limit=0, indicates infinite window size and no retry limit
      b) not providing the args, will default to a)
      c) --autostart-window-size=0 and --autostart-window-retry-limit=<positive value> indicates the autostart process to bail out if the retry limit exceeds irrespective of window period
      d) --autostart-window-size=<x> and --autostart-window-retry-limit=<y> indicates the autostart process to bail out if the retry limit "y" is exceeded for the last window period "x".

            
      Show
      Now one can start hbase services with enabled "autostart/autorestart" feature in controlled fashion with the help of "--autostart-window-size" to define the window period and the "--autostart-window-retry-limit" to define the number of times the hbase services have to be restarted upon being killed/terminated abnormally within the provided window perioid. The following cases are supported with "autostart/autorestart": a) --autostart-window-size=0 and --autostart-window-retry-limit=0, indicates infinite window size and no retry limit b) not providing the args, will default to a) c) --autostart-window-size=0 and --autostart-window-retry-limit=<positive value> indicates the autostart process to bail out if the retry limit exceeds irrespective of window period d) --autostart-window-size=<x> and --autostart-window-retry-limit=<y> indicates the autostart process to bail out if the retry limit "y" is exceeded for the last window period "x".       

    Description

      As part of HBASE-5939, the autorestart for hbase services has been added to deal with scenarios where hbase services (master/regionserver/master-backup) gets killed or goes down leading to unplanned outages. The changes were made to hbase-daemon.sh to support autorestart option.

      However, the autorestart implementation doesn't work in standalone mode and other than that have few gaps with the implementation as per release notes of HBASE-5939. Here is an attempt to re-design and fix the functionality considering all possible usecases with hbase service operations.

      Release Notes of HBASE-5939:
      ------------------------------------------
      When launched with autorestart, HBase processes will automatically restart if they are not properly terminated, either by a "stop" command or by a cluster stop. To ensure that it does not overload the system when the server itself is corrupted and the process cannot be restarted, the server sleeps for 5 minutes before restarting if it was already started 5 minutes ago previously. To use it, launch the process with "bin/start-hbase autorestart". This option is not fully compatible with the existing "restart" command: if you ask for a restart on a server launched with autorestart, the server will restart but the next server instance won't be automatically restarted.

      Attachments

        1. HBASE-15924.master.0001.patch
          19 kB
          Loknath Priyatham Teja Singamsetty
        2. HBASE-15924.master.0002.patch
          21 kB
          Loknath Priyatham Teja Singamsetty
        3. HBASE-15924.master.0003.patch
          21 kB
          Loknath Priyatham Teja Singamsetty
        4. HBASE-15924.master.0004.patch
          22 kB
          Loknath Priyatham Teja Singamsetty

        Issue Links

          Activity

            People

              singamteja Loknath Priyatham Teja Singamsetty
              singamteja Loknath Priyatham Teja Singamsetty
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: