Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20188 Evaluate and address performance delta between branch-1 and branch-2
  3. HBASE-21738

Remove all the CSLM#size operation in our memstore because it's an quite time consuming.

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.
      Show
      We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.

    Description

      Made some performance test for 100% put case in branch-2 before.

      We can see that there are many latency peak in p999 latency curve , and the peak time are almost the point time which our region is flushing.

      See the hbase20-ssd-put-10000000000-rows-latencys-and-qps

      And, I used the add-some-log.patch to log some time consuming when we grab the update.writeLock() to make a memstore snapshot. Tested again, I found those logs in log.txt.

      Seems most of the time was consumed when taking memstore snapshot.. Let me dig into this.

      Attachments

        1. add-some-log.patch
          3 kB
          Zheng Hu
        2. HBASE-21738.v1.patch
          87 kB
          Zheng Hu
        3. HBASE-21738.v2.patch
          49 kB
          Zheng Hu
        4. HBASE-21738.v3.patch
          51 kB
          Zheng Hu
        5. HBASE-21738.v4.patch
          52 kB
          Zheng Hu
        6. image-2019-01-18-14-03-28-662.png
          359 kB
          Zheng Hu
        7. log.txt
          12 kB
          Zheng Hu
        8. performance-after-the-patch.v3.png
          265 kB
          Zheng Hu

        Issue Links

          Activity

            People

              openinx Zheng Hu
              openinx Zheng Hu
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: