Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5017

Bump the default hfile.block.cache.size because of HFileV2

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.92.0
    • 0.92.0, 0.94.0
    • None
    • None
    • Incompatible change, Reviewed
    • The default percentage of heap space allocated to the block cache has been increased from 20% to 25%. However, index caching is now accounted for in the block cache, so total heap usage should be minimally affected.

    Description

      Here's the email I sent to the mailing list describing the problem:

      A thought just stuck me while I was writing down a more detailed block
      caching documentation: with HFileV2, the indexes now live in the block
      cache which means that those who upgrade may all of a sudden get
      terrible cache hit ratios because of all that memory taken by the
      indexes. This is somewhat mitigated by the fact that people don't
      usually need to keep all the index blocks in memory so in the end
      we're more efficient.

      Which brings me to a question: should we set hfile.block.cache.size
      higher since indexes are now kept in the block cache? Currently it's
      set to 20%.

      Looking over my own production machines I see that the
      storefileIndexSize is around 600-700MB so that's potentially how much
      more data I'd have to block cache (more likely it's half of that
      that's really being used actively).

      What would be a good new default? 25%? 30%? How do we handle those
      that will be pushed over the BC+memstore size limit because of that
      change?

      I'll bump this to 25% and put in the release note the fact that people should verify their settings before upgrading to make sure memstore+block cache isn't over 80% (meaning they'd haven't change the block cache size but would have bumped the memstores from 40% to 60%).

      Attachments

        1. HBASE-5017-trunk-v2.patch
          7 kB
          Jean-Daniel Cryans
        2. HBASE-5017-trunk.patch
          5 kB
          Jean-Daniel Cryans
        3. HBASE-5017.patch
          5 kB
          Jean-Daniel Cryans

        Activity

          People

            jdcryans Jean-Daniel Cryans
            jdcryans Jean-Daniel Cryans
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: