Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20188 [TESTING] Performance
  3. HBASE-21738

Remove all the CSLM#size operation in our memstore because it's an quite time consuming.

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0, 2.1.3, 2.0.5
    • Component/s: Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.
      Show
      We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.

      Description

      Made some performance test for 100% put case in branch-2 before.

      We can see that there are many latency peak in p999 latency curve , and the peak time are almost the point time which our region is flushing.

      See the hbase20-ssd-put-10000000000-rows-latencys-and-qps

      And, I used the add-some-log.patch to log some time consuming when we grab the update.writeLock() to make a memstore snapshot. Tested again, I found those logs in log.txt.

      Seems most of the time was consumed when taking memstore snapshot.. Let me dig into this.

        Attachments

        1. add-some-log.patch
          3 kB
          Zheng Hu
        2. HBASE-21738.v1.patch
          87 kB
          Zheng Hu
        3. HBASE-21738.v2.patch
          49 kB
          Zheng Hu
        4. HBASE-21738.v3.patch
          51 kB
          Zheng Hu
        5. HBASE-21738.v4.patch
          52 kB
          Zheng Hu
        6. image-2019-01-18-14-03-28-662.png
          359 kB
          Zheng Hu
        7. log.txt
          12 kB
          Zheng Hu
        8. performance-after-the-patch.v3.png
          265 kB
          Zheng Hu

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: