Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15924

Enhance hbase services autorestart capability to hbase-daemon.sh

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.98.19
    • 1.4.0, 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      Now one can start hbase services with enabled "autostart/autorestart" feature in controlled fashion with the help of "--autostart-window-size" to define the window period and the "--autostart-window-retry-limit" to define the number of times the hbase services have to be restarted upon being killed/terminated abnormally within the provided window perioid.

      The following cases are supported with "autostart/autorestart":

      a) --autostart-window-size=0 and --autostart-window-retry-limit=0, indicates infinite window size and no retry limit
      b) not providing the args, will default to a)
      c) --autostart-window-size=0 and --autostart-window-retry-limit=<positive value> indicates the autostart process to bail out if the retry limit exceeds irrespective of window period
      d) --autostart-window-size=<x> and --autostart-window-retry-limit=<y> indicates the autostart process to bail out if the retry limit "y" is exceeded for the last window period "x".

            
      Show
      Now one can start hbase services with enabled "autostart/autorestart" feature in controlled fashion with the help of "--autostart-window-size" to define the window period and the "--autostart-window-retry-limit" to define the number of times the hbase services have to be restarted upon being killed/terminated abnormally within the provided window perioid. The following cases are supported with "autostart/autorestart": a) --autostart-window-size=0 and --autostart-window-retry-limit=0, indicates infinite window size and no retry limit b) not providing the args, will default to a) c) --autostart-window-size=0 and --autostart-window-retry-limit=<positive value> indicates the autostart process to bail out if the retry limit exceeds irrespective of window period d) --autostart-window-size=<x> and --autostart-window-retry-limit=<y> indicates the autostart process to bail out if the retry limit "y" is exceeded for the last window period "x".       

    Description

      As part of HBASE-5939, the autorestart for hbase services has been added to deal with scenarios where hbase services (master/regionserver/master-backup) gets killed or goes down leading to unplanned outages. The changes were made to hbase-daemon.sh to support autorestart option.

      However, the autorestart implementation doesn't work in standalone mode and other than that have few gaps with the implementation as per release notes of HBASE-5939. Here is an attempt to re-design and fix the functionality considering all possible usecases with hbase service operations.

      Release Notes of HBASE-5939:
      ------------------------------------------
      When launched with autorestart, HBase processes will automatically restart if they are not properly terminated, either by a "stop" command or by a cluster stop. To ensure that it does not overload the system when the server itself is corrupted and the process cannot be restarted, the server sleeps for 5 minutes before restarting if it was already started 5 minutes ago previously. To use it, launch the process with "bin/start-hbase autorestart". This option is not fully compatible with the existing "restart" command: if you ask for a restart on a server launched with autorestart, the server will restart but the next server instance won't be automatically restarted.

      Attachments

        1. HBASE-15924.master.0001.patch
          19 kB
          Loknath Priyatham Teja Singamsetty
        2. HBASE-15924.master.0002.patch
          21 kB
          Loknath Priyatham Teja Singamsetty
        3. HBASE-15924.master.0003.patch
          21 kB
          Loknath Priyatham Teja Singamsetty
        4. HBASE-15924.master.0004.patch
          22 kB
          Loknath Priyatham Teja Singamsetty

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            singamteja Loknath Priyatham Teja Singamsetty
            singamteja Loknath Priyatham Teja Singamsetty
            Votes:
            0 Vote for this issue
            Watchers:
            7 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment