Three Layers Three Attempts

Each layer of a request path retries twice on a 5xx, so each makes up to 3 attempts. The client SDK calls the gateway, the gateway calls the checkout service through its sidecar, and checkout calls the ledger.

# client SDK
retry: { max_attempts: 3 }
# gateway route
retry_policy: { retry_on: 5xx, num_retries: 2 }
# checkout sidecar
retry_policy: { retry_on: 5xx, num_retries: 2 }

The ledger starts returning 503. How many requests can one user action put on the ledger, and what is the standard fix?

7, one initial attempt plus two retries at each of the three layers

$3^3 = 27$ , and the fix is to retry at one layer only, with a service-wide retry budget

$2^3 = 8$ , and the fix is exponential backoff at each layer

$3 \times 3 = 9$ , since the client SDK cannot retry a request the gateway already failed