Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Fixed
-
None
-
Availability
-
Normal
-
Normal
-
User Report
-
All
-
None
-
Description
When a node is under pressure, hundreds of thousands of requests can show up in the native transport queue, and it looks like it can take way longer to timeout than is configured. We should be shedding load much more aggressively and use a bounded queue for incoming work. This is extremely evident when we combine a resource consuming workload with a smaller one:
Running 5.0 HEAD on a single node as of today:
# populate only easy-cass-stress run RandomPartitionAccess -p 100 -r 1 --workload.rows=100000 --workload.select=partition --maxrlat 100 --populate 10m --rate 50k -n 1 # workload 1 - larger reads easy-cass-stress run RandomPartitionAccess -p 100 -r 1 --workload.rows=100000 --workload.select=partition --rate 200 -d 1d # second workload - small reads easy-cass-stress run KeyValue -p 1m --rate 20k -r .5 -d 24h
It appears our results don't time out at the requested server time either:
Writes Reads Deletes Errors Count Latency (p99) 1min (req/s) | Count Latency (p99) 1min (req/s) | Count Latency (p99) 1min (req/s) | Count 1min (errors/s) 950286 70403.93 634.77 | 789524 70442.07 426.02 | 0 0 0 | 9580484 18980.45 952304 70567.62 640.1 | 791072 70634.34 428.36 | 0 0 0 | 9636658 18969.54 953146 70767.34 640.1 | 791400 70767.76 428.36 | 0 0 0 | 9695272 18969.54 956833 71171.28 623.14 | 794009 71175.6 412.79 | 0 0 0 | 9749377 19002.44 959627 71312.58 656.93 | 795703 71349.87 435.56 | 0 0 0 | 9804907 18943.11
After stopping the load test altogether, it took nearly a minute before the requests were no longer queued.
Attachments
Attachments
Issue Links
- supercedes
-
-
- Resolved
-
- links to
