Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10367

RegionServer graceful stop / decommissioning

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0-alpha-4, 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      Added three top level Admin APIs to help decommissioning and graceful stop of region servers.

        /**
         * Mark region server(s) as decommissioned to prevent additional regions from getting
         * assigned to them. Optionally unload the regions on the servers. If there are multiple servers
         * to be decommissioned, decommissioning them at the same time can prevent wasteful region
         * movements. Region unloading is asynchronous.
         * @param servers The list of servers to decommission.
         * @param offload True to offload the regions from the decommissioned servers
         */
        void decommissionRegionServers(List<ServerName> servers, boolean offload) throws IOException;

        /**
         * List region servers marked as decommissioned, which can not be assigned regions.
         * @return List of decommissioned region servers.
         */
        List<ServerName> listDecommissionedRegionServers() throws IOException;

        /**
         * Remove decommission marker from a region server to allow regions assignments.
         * Load regions onto the server if a list of regions is given. Region loading is
         * asynchronous.
         * @param server The server to recommission.
         * @param encodedRegionNames Regions to load onto the server.
         */
        void recommissionRegionServer(ServerName server, List<byte[]> encodedRegionNames) throws IOException;

      Show
      Added three top level Admin APIs to help decommissioning and graceful stop of region servers.   /**    * Mark region server(s) as decommissioned to prevent additional regions from getting    * assigned to them. Optionally unload the regions on the servers. If there are multiple servers    * to be decommissioned, decommissioning them at the same time can prevent wasteful region    * movements. Region unloading is asynchronous.    * @param servers The list of servers to decommission.    * @param offload True to offload the regions from the decommissioned servers    */   void decommissionRegionServers(List<ServerName> servers, boolean offload) throws IOException;   /**    * List region servers marked as decommissioned, which can not be assigned regions.    * @return List of decommissioned region servers.    */   List<ServerName> listDecommissionedRegionServers() throws IOException;   /**    * Remove decommission marker from a region server to allow regions assignments.    * Load regions onto the server if a list of regions is given. Region loading is    * asynchronous.    * @param server The server to recommission.    * @param encodedRegionNames Regions to load onto the server.    */   void recommissionRegionServer(ServerName server, List<byte[]> encodedRegionNames) throws IOException;

    Description

      Right now, we have a weird way of node decommissioning / graceful stop, which is a graceful_stop.sh bash script, and a region_mover ruby script, and some draining server support which you have to manually write to a znode (really!). Also draining servers is only partially supported in LB operations (LB does take that into account for roundRobin assignment, but not for normal balance)
      See
      http://hbase.apache.org/book/node.management.html and HBASE-3071

      I think we should support graceful stop as a first class citizen. Thinking about it, it seems that the difference between regionserver stop and graceful stop is that regionserver stop will close the regions, but the master will only assign them after the znode is deleted.

      In the new master design (or even before), if we allow RS to be able to close regions on its own (without master initiating it), then graceful stop becomes regular stop. The RS already closes the regions cleanly, and will reject new region assignments, so that we don't need much of the balancer or draining server trickery.

      This ties into the new master/AM redesign (HBASE-5487), but still deserves it's own jira. Let's use this to brainstorm on the design.

      Attachments

        1. HBASE-10367-master-2.patch
          78 kB
          Jerry He
        2. HBASE-10367-master.patch
          77 kB
          Jerry He
        3. HBASE-10367-master.patch
          77 kB
          Michael Stack

        Issue Links

          Activity

            People

              jinghe Jerry He
              enis Enis Soztutar
              Votes:
              1 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: