[MAPREDUCE-3787] [Gridmix] Improve STRESS mode - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.0.0-alpha
Fix Version/s: 2.1.0-beta
Component/s: contrib/gridmix
Labels:
- gridmix
- stress

Target Version/s:
Hadoop Flags:

Reviewed
Release Note:

Hide
JobMonitor can now deploy multiple threads for faster job-status polling. Use 'gridmix.job-monitor.thread-count' to set the number of threads. Stress mode now relies on the updates from the job monitor instead of polling for job status. Failures in job submission now get reported to the statistics module and ultimately reported to the user via summary.

Show
JobMonitor can now deploy multiple threads for faster job-status polling. Use 'gridmix.job-monitor.thread-count' to set the number of threads. Stress mode now relies on the updates from the job monitor instead of polling for job status. Failures in job submission now get reported to the statistics module and ultimately reported to the user via summary.
Tags:
gridmix3 stress

Description

Gridmix STRESS mode can be improved as follows:
1. The sleep time in JobMonitor can be reduced and/or made configurable
2. Map and reduce load calculation in StressJobFactory can be done in one loop
3. Updating the overload status from the job submitter thread (inline)
4. Optimizations to avoid un-necessary progress check (which inturn would result into delay)