4 / 6 The Tradeoff Vocabulary

Twelve Attempts Per Request

Retries were added independently at three layers, each by a different team, each reasonably.

Layer Retry policy
Mobile client up to 3 attempts
API gateway up to 2 attempts per client attempt
Order service to database up to 2 attempts per gateway attempt

The database has a 40 second failover blip. Normal load is 900 QPS. What arrives at the database during the blip, and what is the right correction?

A

Up to 12 attempts per user request. Raise the timeout at each layer so fewer calls are classified as failures in the first place.

B

Up to 12 attempts per user request, roughly 10,800 QPS. Retry at one layer only, the one that can tell the failure is retriable, and cap total retries with a token bucket so they stay a small fraction of normal traffic.

C

Up to 7 attempts per user request. Add exponential backoff with jitter at all three layers so the attempts spread out over the blip.

D

Up to 12 attempts per user request. Raise the database connection pool and instance size so the amplified rate fits.