2 / 6 Latency, Throughput & The Numbers

The Last Ten Percent

A service processes each request in a steady 10 ms of actual work. As traffic grew, utilization went from 50 percent to 90 percent and the measured average response time went from about 20 ms to about 100 ms, with no code change in between. What is happening?

A

Cache pressure at high load is slowing every request by roughly the same amount, which is the usual cause of average response time growth in a service running this hot.

B

Queueing delay dominates as utilization rises: response time scales roughly as service time divided by one minus utilization, so the last points of utilization are the expensive ones.

C

Response time grows in proportion to utilization, so going from 50 to 90 percent should raise the average by about 1.8 times, and the rest of the fivefold rise needs a separate explanation.

D

Per-request work has silently grown somewhere in the dependency chain, because a service whose service time is constant cannot change its response time from load alone.