5 / 5 Load Balancing

The Fastest Failure Wins

A client library balances with least-request: send the next request to the backend with the fewest in-flight requests. One backend's dependency is broken and it returns 503 almost instantly.

backend response p50 latency in-flight, sampled over 1 s
A 200 120 ms 12
B 200 118 ms 12
C 503 4 ms 0 to 1
D 200 125 ms 13

What happens to the traffic split over the next minute?

A

C takes about half, the share it would get if the three healthy backends were each twice as slow

B

C takes the large majority of new requests, because its in-flight count falls back to zero every 4 ms and it therefore always looks least loaded

C

The split stays near even at one quarter each, since in-flight counts only break ties between otherwise equal backends

D

C receives nothing, because a backend returning 503 fails its health check and is pulled from the pool