Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21428

Performance issue due to userRegionLock in the ConnectionManager.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 1.2.7
    • None
    • None
    • None

    Description

      My service is that execute a lot of puts using HTableMultiplexer.
      After the version change, most of the requests are rejected.

      It works fine in 1.2.6.1, but there is a problem in 1.2.7.

      This issue is related with the HBASE-19260.

      Most of my threads are using a lot of time as below.

       

      "Worker-972" #2479 daemon prio=5 os_prio=0 tid=0x00007f8cea86b000 nid=0x4c8c waiting on condition [0x00007f8b78104000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000005dd703b78> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1274)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1186)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1170)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1127)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:962)
        at org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:206)
        at org.apache.hadoop.hbase.client.HTableMultiplexer.put(HTableMultiplexer.java:150)

       

      When I looked at the issue(HBASE-19260), I recognized the dangerous of to allow accessessing multiple threads.
      However, Already create many threads with the limitations
      I think it is very inefficient to allow only one thread access.

       

       this.metaLookupPool = getThreadPool(
      conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128),
      conf.getInt("hbase.hconnection.meta.lookup.threads.core", 10),
      "metaLookup-shared", new LinkedBlockingQueue<Runnable>());

       

      I want to suggest changing it that allow to have multiple locks.(but not the entire thread)

      The following is pseudocode.

       

      int lockSize = conf.getInt("hbase.hconnection.meta.lookup.threads.max", 128) / 2;
      BlockingQueue<ReentrantLock> userRegionLockQueue = new LinkedBlockingQueue<ReentrantLock>();
      for (int i=0; i <lockSize; i++) { userRegionLockQueue.put(new ReentrantLock()); }

       

      thanks.

       

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21196 HTableMultiplexer clears the meta cache after every put operation

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              taejin koo
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: