[HBASE-21738] Remove all the CSLM#size operation in our memstore because it's an quite time consuming. - ASF JIRA

Voters

Stop watching

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.3, 2.0.5
Component/s: Performance
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.

Show
We found the memstore snapshotting would cost much time because of calling the time-consuming ConcurrentSkipListMap#Size, it would make the p999 latency spike happen. So in this issue, we remove all ConcurrentSkipListMap#size in memstore by counting the cellsCount in MemstoreSizeing. As the issue described, the p999 latency spike was mitigated.

Description

Made some performance test for 100% put case in branch-2 before.

We can see that there are many latency peak in p999 latency curve , and the peak time are almost the point time which our region is flushing.

See the hbase20-ssd-put-10000000000-rows-latencys-and-qps

And, I used the add-some-log.patch to log some time consuming when we grab the update.writeLock() to make a memstore snapshot. Tested again, I found those logs in log.txt.

Seems most of the time was consumed when taking memstore snapshot.. Let me dig into this.