Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20828 Finish-up AMv2 Design/List of Tenets/Specification of operation
  3. HBASE-21423

Procedures for meta table/region should be able to execute in separate workers

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1, 2.0.2
    • Fix Version/s: 2.0.3, 2.1.2
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0.
      Show
      The procedure for meta table will be executed in a separate worker thread named 'Urgent Worker' to avoid stuck. A new config named 'hbase.master.urgent.procedure.threads' is added, the default value for it is 1. To disable the separate worker, set it to 0.

      Description

      We have higher priority for meta table procedures, but only in queue level. There is a case that the meta table is closed and a AssignProcedure(or RTSP in branch-2+) is waiting there to be executed, but at the same time, all the Work threads are executing procedures need to write to meta table, then all the worker will be stuck and retry for writing meta, no worker will take the AP for meta.
      Though we have a mechanism that will detect stuck and adding more ''KeepAlive'' workers to the pool to resolve the stuck. It is already stuck a long time.
      This is a real case I encountered in ITBLL.
      So, I add one 'Urgent work' to the ProceudureExecutor, which only take meta procedures(other workers can take meta procedures too), which can resolve this kind of stuck.

        Attachments

        1. HBASE-21423.branch-2.0.001.patch
          20 kB
          Allan Yang
        2. HBASE-21423.branch-2.0.002.patch
          29 kB
          Allan Yang
        3. HBASE-21423.branch-2.0.003.patch
          30 kB
          Allan Yang
        4. HBASE-21423.branch-2.0.addendum.patch
          16 kB
          Allan Yang

          Issue Links

            Activity

              People

              • Assignee:
                allan163 Allan Yang
                Reporter:
                allan163 Allan Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: