change the random scope of random row for PerformanceEvaluation

question description:

when we use PerformanceEvaluation tool to run randomWrite test, we find out that when one region happened regiontoobusy, the requests in the ui will dramatically decreased in a short time from several million to 0. 

the mechanism of regiontoobusy is really nice to maximum the throughput of hbase cluster, it will only influence the current region which happened regiontoobusy. but when i look into the whole procedure of the random write(include client and server), i found there may be some problem in PerformanceEvaluation tool(client) which cause the current question.

 

cause of the issue:

before trying to illustrate the issue, here are some preconditions we need to know first:

1、one request generated by TestClient thread will contain the data of all regions(we will accumulate 2M(default client buffer size) request, the request will include many mutate operations. each mutate operation is generated in random from a whole table)

2、when one 2M's request doesn't finish, the TestClient won't generate a new request(due to the PE's mechanism)

let's try to illustrate the cause of the issue:

1、when one region happened regiontoobusy, the 2M's request will not be finished util the region unblocked. so do the other 2M's request generated by other TestClients.

2、both the client(PerformanceEvaluation) and server(regionserver) will block and we will see the request in the ui and cpu util decreased in several seconds util the unblock of the region.

probable solution：

the issue is not due to regionserver(regiontoobusy is a good mechanism from the regionserver's side), but the client. 

if we try to change the scope of random, we will find a way to solve this problem.

the origin scope is the whole table, if every TestClient generate random row in it's own scope we will solve this problem, let's take a example:

assume that we have 5 TestClient, each TestClient charge for 1000 requests.

TestClient1 will generate random rows from 0–999;

TestClient2 will generate random rows from 1000–1999;

TestClient3 will generate random rows from 2000–2999;

TestClient4 will generate random rows from 3000--3999;

TestClient5 will generate random rows from 4000–4999;

 

some other words to say:

i raise a question encountered in my work. i hope to have a further discussion with the experts from the community and other fields in order to seek a better solution. if my solution is acceptable, i will push a patch to solve the issue.