Mid Tier Thundering Herd Overwhelms Origin when Origin Slow to Respond

Mid tier groups typically send one request per object to origin. We have noticed that when an origin is slow to respond or sends errors that the mid tier sometimes sends many requests at a high TPS rate which causes a slow origin to fail.

We are investigating this on both the ATS and ATC side. We are considering tuning the mid tier retry attempts as well as some other configuration settings to reduce the potential for overload.