6 / 6 Timeouts, Retries & Circuit Breakers

The Blip Is Over

A 30s network partition caused widespread timeouts. The partition has been resolved for ten minutes, inbound user traffic is back at its normal daytime level, yet the service is pinned at 100% CPU with almost zero successful responses. Blocking all inbound traffic at the edge for 60 seconds and then re-admitting it restores the service instantly, with no restart and no deploy. What is the best explanation?

A

The fleet was already undersized for normal daytime traffic, and the partition revealed a shortfall that had been masked by lighter load

B

A sustaining feedback loop: timed-out work is being retried, and the retry load alone now exceeds capacity, so the system cannot recover on its own

C

The partition is not actually resolved, and the monitoring reporting it healthy probes a different network path than real traffic takes

D

A memory leak triggered during the partition has filled the heap, so the fleet now spends its CPU in garbage collection until a restart reclaims it