Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23223

Support the offsetLock of bucketCache to use strong reference

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
    • 3.0.0-alpha-1
    • BucketCache
    • None
    • Hide
      The offsetLock of bucketCache use soft reference based impl by default,it could reclaim mem when not enough,but a lot of soft reference object would lead to long puase when mixed gc occurs,so if the count of offset is limited and you have enough mem(most of the time it is true),then the strong reference is better choice.

      Set hbase.bucketcache.offsetlock.usestrongref=true to enable it.
      Show
      The offsetLock of bucketCache use soft reference based impl by default,it could reclaim mem when not enough,but a lot of soft reference object would lead to long puase when mixed gc occurs,so if the count of offset is limited and you have enough mem(most of the time it is true),then the strong reference is better choice. Set hbase.bucketcache.offsetlock.usestrongref=true to enable it.

    Description

      My live cluster env config below:
      hbase version: cdh6.0.1(apache hbase2.0.0)
      hbase config: bucketCache(70g),blocksize(16k)
      java version: 1.8.0_51
      java config: heap(32g),-XX:+UseG1GC  -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled

       

      The offsetLock of BucketCache use SoftReference for its ReentrantReadWriteLock,when bucketCache is big,it will cost too much time in Gc ref-proc.

      eg : 
      2019-09-29T01:55:45.186+0800: 365222.053:
      [GC remark
      2019-09-29T01:55:45.186+0800: 365222.053:
      [Finalize Marking, 0.0016327 secs]
      2019-09-29T01:55:45.188+0800: 365222.054:
      [GC ref-proc
      2019-09-29T01:55:45.188+0800: 365222.054: [SoftReference, 1264586 refs, 0.3151392 secs]
      2019-09-29T01:55:45.503+0800: 365222.370: [WeakReference, 4317 refs, 0.0024381 secs]
      2019-09-29T01:55:45.505+0800: 365222.372: [FinalReference, 9791 refs, 0.0037445 secs]
      2019-09-29T01:55:45.509+0800: 365222.376: [PhantomReference, 0 refs, 1963 refs, 0.0018941 secs]
      2019-09-29T01:55:45.511+0800: 365222.378: [JNI Weak Reference, 0.0001156 secs]
      , 1.4554361 secs]
      2019-09-29T01:55:46.643+0800: 365223.510:
      [Unloading, 0.0211370 secs]
      , 1.4851728 secs]

       
      I think the strong reference is a better choice in some case ,so i changed it and get good effect on my live cluster:
       
      2019-10-26T02:29:10.248+0800: 131255.447:
      [GC remark
      2019-10-26T02:29:10.248+0800: 131255.447:
      [Finalize Marking, 0.0011426 secs]
      2019-10-26T02:29:10.249+0800: 131255.448:
      [GC ref-proc
      2019-10-26T02:29:10.249+0800: 131255.448: [SoftReference, 213 refs, 0.0002216 secs]
      2019-10-26T02:29:10.249+0800: 131255.448: [WeakReference, 2423 refs, 0.0007833 secs]
      2019-10-26T02:29:10.250+0800: 131255.449: [FinalReference, 4991 refs, 0.0061449 secs]
      2019-10-26T02:29:10.256+0800: 131255.455: [PhantomReference, 0 refs, 7940 refs, 0.0011711 secs]
      2019-10-26T02:29:10.257+0800: 131255.456: [JNI Weak Reference, 0.0000579 secs]
      , 0.0086389 secs]
      2019-10-26T02:29:10.257+0800: 131255.456:
      [Unloading, 0.0123417 secs]
      , 0.0294484 secs]
       
      Strong reference will always hold the mem,so it maybe not fit for some case that have not enough mem, so i make it configable.

       

      Attachments

        1. HBASE-23223-benchmark.pdf
          243 kB
          Zheng Wang
        2. HBASE-23223-RS metrics with patch.pdf
          139 kB
          Zheng Wang

        Issue Links

          links to

          Web Link GitHub Pull Request #764

          Activity

            People

              filtertip Zheng Wang
              filtertip Zheng Wang
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: