7 / 7 Back-of-the-Envelope Estimation

In Flight At Once

A service handles 5,000 QPS with an average server-side latency of 200 ms. Roughly how many requests are in flight at any instant, and why does the number matter?

A

About 1,000,000, since each of the 5,000 arrivals per second holds a worker slot for 200 units of time, and that product is what the pools must cover

B

About 1,000, since concurrency equals arrival rate times latency, so thread and connection pools across the fleet must total at least that

C

About 5,000, since every request that arrives during a given second is in flight at some point in that second, so the pools must cover a second of arrivals

D

About 25,000, since dividing the 5,000 per second arrival rate by the 0.2 second service time gives the number of requests resident in the system at once