Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9642

AM ZK Workers stuck doing 100% CPU on HashMap.put

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.96.0
    • 0.98.0, 0.96.0
    • None
    • None

    Description

      I just noticed on my test cluster that my master is using all my CPUs even though it's completely idle. 5 threads are doing this:

      "AM.ZK.Worker-pool2-t34" daemon prio=10 tid=0x00007f68ac176800 nid=0x5251 runnable [0x00007f688cc83000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.HashMap.put(HashMap.java:374)
      	at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:954)
      	at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1419)
      	at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1247)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      

      Looking at the code, I see HBASE-9095 introduced two HashMaps for tests only but they end up being used concurrently in the AM and are never cleaned up. It seems to me that any master running since that patch was committed has a time bomb in it.

      I'm marking this as a blocker. devaraj and jxiang, you guys wanna take a look at this?

      Attachments

        1. 9642-1.txt
          5 kB
          Devaraj Das
        2. 9642-2.txt
          5 kB
          Devaraj Das

        Activity

          People

            ddas Devaraj Das
            jdcryans Jean-Daniel Cryans
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: