Twelve Attempts Per Request

Retries were added independently at three layers, each by a different team, each reasonably.

Layer	Retry policy
Mobile client	up to 3 attempts
API gateway	up to 2 attempts per client attempt
Order service to database	up to 2 attempts per gateway attempt

The database has a 40 second failover blip. Normal load is 900 QPS. What arrives at the database during the blip, and what is the right correction?

Up to 12 attempts per user request. Raise the timeout at each layer so fewer calls are classified as failures in the first place.

Up to 12 attempts per user request, roughly 10,800 QPS. Retry at one layer only, the one that can tell the failure is retriable, and cap total retries with a token bucket so they stay a small fraction of normal traffic.

Up to 7 attempts per user request. Add exponential backoff with jitter at all three layers so the attempts spread out over the blip.

Up to 12 attempts per user request. Raise the database connection pool and instance size so the amplified rate fits.