Cache Falls To Database

A retail site shows per-product shipping speeds, sometimes 25 of them on one page, read from a supply chain database. Because the database cannot carry that read rate, each web server runs a local cache process in front of it. The code reads:

if (cache_healthy) {
    speed = get_speed_via_cache(sku);
} else {
    speed = get_speed_from_database(sku);
}

This works for months. Then the cache processes fail across the fleet at roughly the same time, every web server goes straight to the database, the database locks up, and the entire site goes down along with the fulfilment network that shares that database. What does this incident reveal about the design?

The cache read should have been retried with exponential backoff and jitter before falling through to the database.

The fallback path could never carry the load the cache existed to absorb, so it was not a substitute at all. It was a switch that converted a broken shipping-speed widget into a whole-site outage.

The per-host cache was the mistake. A shared cache cluster would not have failed across every host at once.

The fallback needed a circuit breaker, so after a threshold of failed database calls the servers would stop calling it and protect the database.