Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-25697

StochasticBalancer improvement for large scale clusters

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Balancer, master, UI
    • None

    Description

      Findings on a large scale cluster (100,000 regions on 300 nodes)

      • Balancer starts and stops before getting a plan
      • Adding new racks doesn’t trigger balancer
      • Balancer stops leaving some racks at 50% lower region counts
      • Regions for large tables don’t get evenly distributed
      • Observability is poor
      • Too many knobs makes tuning empirical and takes many experiments

      Improvements made and being made

      More proposals

      • minCostNeedBalance for each cost function instead of weights. We want to trigger balancing if any factor is out of balancer instead of trying to combine the factors in arbitrary weights. This makes operation and configuration much easier.
      • Simulated annealing to lower minCostNeedBalance periodically to unstuck the balancer from sub-optimum then gradually increase to keep the system stable. Also add cost of move as a counter measure for the decision https://opensourcelibs.com/lib/tempest
      • Orchestrated scheduling of compaction, normalizer and balancer
      • PID approach https://www.amazon.com/dp/1449361692/ref=rdr_ext_tmb

      Attachments

        Issue Links

          is a parent of

          Sub-task - The sub-task of the issue HBASE-26308 Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Sub-task - The sub-task of the issue HBASE-26310 Repro balancer behavior during iterations

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HBASE-26237 Improve computation complexity for primaryRegionCountSkewCostFunctio

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-26177 Add support to run balancer overriding current config

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HBASE-27302 Adding a trigger for Stochastica Balancer to safeguard for upper bound outliers.

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HBASE-26178 Improve data structure and algorithm for BalanceClusterState to improve computation speed for large cluster

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-26297 Balancer run is improperly triggered by accuracy error of double comparison

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-26311 Balancer gets stuck in cohosted replica distribution

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-24643 Replace Cluster#primariesOfRegionsPerServer from int array to treemap

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-25625 StochasticBalancer CostFunctions needs a better way to evaluate region count distribution

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-25666 Explain why balancer is skipping runs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-26309 Balancer tends to move regions to the server at the end of list

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-26337 Optimization for weighted random generators

          • Major - Major loss of function.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HBASE-24528 Improve balancer decision observability

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-25666 Explain why balancer is skipping runs

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-26147 Add dry run mode to hbase balancer

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              claraxiong Clara Xiong
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: