Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21200

Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0, 2.1.1, 2.0.3
    • Component/s: Scanners
    • Labels:
      None

      Description

      The  issue of delaying memstore flush still occurs after backport hbase-15871.

      Reverse scan takes a long time to seek previous row in the memstore full of deleted cells.

       

      jstack :

      "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x00007fa3d0729000 nid=0x486a waiting on condition [0x00007fa3b9b6b000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000000a465fe60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:695)
              at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127)
              at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106)
              at org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130)
              at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218)
              at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110)
              at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
              at java.lang.Thread.run(Thread.java:748)

       

      "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5 os_prio=0 tid=0x00007fa3e6280000 nid=0x4801 runnable [0x00007fa3bd29a000]
         java.lang.Thread.State: RUNNABLE
              at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780)
              at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826)
              - locked <0x00000000b45aa5b8> (a org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
              at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818)
              - locked <0x00000000b45aa5b8> (a org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
              at org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000)
              - locked <0x00000000b45aa5b8> (a org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner)
              at org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136)
              at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:629)
              at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
              - locked <0x000000009ee8ca10> (a org.apache.hadoop.hbase.regionserver.ReversedRegionScannerImpl)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)

        Attachments

        1. HBASE-21200.master.001.patch
          4 kB
          Toshihiro Suzuki
        2. HBASE-21200.master.002.patch
          5 kB
          Toshihiro Suzuki
        3. HBASE-21200-UT.patch
          2 kB
          Toshihiro Suzuki
        4. RegionServerJstack.log
          11.00 MB
          dongjin2193.jeon

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                brfrn169 Toshihiro Suzuki
                Reporter:
                dongjin2193 dongjin2193.jeon
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: