Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21355

HStore's storeSize is calculated repeatedly which causing the confusing region split

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      When testing the branch-2's write performance in our internal cluster, we found that the region will be inexplicably split.

      We use the default ConstantSizeRegionSplitPolicy and hbase.hregion.max.filesize=40G,but the region will be split even if its bytes size is less than 40G(only ~6G).

      Checked the code, I found that the following path will accumulate the store's storeSize to a very big value, because the path has no reset..

      RsRpcServices#getRegionInfo
            -> HRegion#isMergeable
                 -> HRegion#hasReferences
                      -> HStore#hasReferences
                              -> HStore#openStoreFiles
      

      BTW, we seems forget to maintain the read replica's storeSize when refresh the store files.

      Some comment here, I move the storeSize calculation out of loadStoreFiles() method, because the secondary read replica's refreshStoreFiles() will also use loadStoreFiles() to refresh its store files and update the storeSize in the completeCompaction(..) in the final (just like compaction.) , so no need calculate the storeSize twice..

      Attachments

        1. HBASE-21355.v1.patch
          6 kB
          Zheng Hu
        2. HBASE-21355.branch-1.patch
          6 kB
          Zheng Hu
        3. HBASE-21355.addendum.patch
          2 kB
          Zheng Hu
        4. HBASE-21355.addendum.patch
          3 kB
          Zheng Hu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            openinx Zheng Hu
            openinx Zheng Hu
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment