Latency measures elapsed time along a request’s dependencies
Latency is the elapsed time between a defined start and finish. For a browser request, the interval might run from sending the request to receiving a usable response. For a database query, it might run from arrival at the database to the production of result rows. These measurements cover different work, so their boundaries belong beside their values.
Elapsed time includes both execution and waiting. A request may wait for a network round trip, a free application worker or a database lock before useful computation begins. Optimizing a small function cannot remove time spent waiting elsewhere. A trace separates these intervals and shows which work depends on an earlier result.
A request’s critical path is the longest required chain of dependent work. Sequential intervals add along that chain. Independent operations can overlap, but the response still waits for the last required result. This dependency structure explains which optimization can reduce the user’s wait and which merely makes an already-short branch finish earlier.
Use one unit when comparing intervals. A millisecond (ms) is one thousandth of a second and equals 1,000 microseconds (µs); a microsecond equals 1,000 nanoseconds (ns). The operations a system performs span all three units, from a nanosecond for a cache reference to a hundred milliseconds or more for a message crossing an ocean, and knowing roughly where each one falls tells you which waits are worth removing.
Latency numbers to remember
| Operation | Typical time | Where the time goes |
|---|---|---|
| L1 cache reference | about 1 ns | A few CPU cycles; the data is already on the core |
| Main memory (DRAM) reference | about 100 ns | A cache miss travels to memory chips outside the core |
| Random 4 KB read from a datacenter NVMe SSD | about 60 to 100 µs | Reading a flash page plus the storage stack; about a thousand memory references |
| Read 1 MB sequentially from an NVMe SSD | about 150 to 200 µs | Throughput-bound: current datacenter drives read 5 to 7 GB per second |
| Round trip within one datacenter | about 0.5 ms or less | Switches and the network stacks on both machines; the distance is short |
| Round trip between zones in one cloud region | single-digit ms | AWS places its availability zones up to about 100 km apart, still close enough for synchronous replication |
| Random read from a 7,200 rpm hard disk | about 5 to 10 ms | Moving the head to the track, then waiting for the sector to rotate under it (half a turn is 4.2 ms) |
| Round trip New York to London over fiber | at least 56 ms | 5,585 km each way at about 200,000 km per second, the speed of light in glass |
| Round trip California to the Netherlands | about 150 ms | Real routes are longer than the great circle and pass through many routers |
Read the table as ratios. A memory reference costs about a hundred L1 references; a random SSD read costs about a thousand memory references; a round trip across a datacenter costs several SSD reads; and a round trip between continents costs hundreds of datacenter round trips. A page that makes 20 sequential calls to a region 150 ms away spends about 3 seconds waiting before any of its own work counts.
Latency is delay; bandwidth is volume per second
Latency is how long one message takes to arrive. Bandwidth is how much data a path can carry per second. The time to deliver a response is roughly the round-trip latency plus the payload size divided by the bandwidth. Sending 1 MB over a 1 Gbps link takes 8 ms of transmission; over a 10 Gbps link it takes 0.8 ms. Between New York and London, the 56 ms or more of propagation dominates either way.
Small requests are therefore latency-bound and large transfers bandwidth-bound. Buying a faster link speeds up a video download but leaves a chatty page that makes many small sequential calls almost unchanged. A new TCP connection adds further round trips for a large first response, because the sender increases its sending rate gradually while it learns the path’s capacity.
Distance sets a floor that no hardware removes. Light in optical fiber travels about 200,000 km per second, so every 100 km of fiber adds about 1 ms to a round trip. Reducing that cost means moving data closer to its readers, as a CDN does, or making fewer sequential round trips.
Add dependent intervals and take the maximum of independent branches
Two operations are serial when the second starts after the first completes. A real dependency may require this order, or the program may introduce it by awaiting one independent result before starting another. In either case, the elapsed intervals add. A request with 40 ms of network travel, 5 ms of queueing, 10 ms of application work, a 25 ms database read of the user’s profile and a 60 ms call to a recommendation service takes 140 ms when all five occur in sequence.
Suppose both downstream calls need only the authenticated user identifier. They can start together after the first 55 ms. The profile read finishes at 80 ms and recommendations at 115 ms, so a response needing both completes at 115 ms. The 25 ms saving comes from overlapping the entire profile read with part of the recommendation call.
Issuing both calls together cannot remove a true data dependency. If recommendations require preferences returned by the profile call, they must wait. Concurrency can also change resource contention: more simultaneous requests may exhaust connection pools or lengthen database queues, so measure whether the downstream durations still hold under the new load.
A response contract can sometimes remove a dependency from the initial critical path. For example, a page may return the profile first and load optional recommendations afterward. That changes what the first response promises and what the user sees when recommendations fail. The choice is useful only if the partial result still serves the user’s task.
Overlap branches, then move the slow dependency
How much does running the two independent calls in parallel save, and which branch then limits the response? The second experiment asks how one slow request among twenty moves the mean, p95 and p99.
Fixed-duration model: one network RTT, one queue and application interval, then two required branches. No transfer, retries or contention are simulated.
Response time: 140 ms
Time overlapped: 0 ms
In parallel, the default request drops from 140 ms to 115 ms; raising the database read to 100 ms makes it the slower branch and the total 155 ms. For nineteen 40 ms requests and one 400 ms request, the mean is 58 ms, p50 and p95 are 40 ms, and p99 is 400 ms.
A latency percentile describes a population of completed observations
A service receives many requests, and their latencies form a distribution. Tail latency describes the slower observations in that distribution, usually reported as the 99th or 99.9th percentile. Slow requests may reflect lock contention, a cold cache, garbage collection or a delayed dependency even when the ordinary request is fast. A single average cannot reveal how many callers wait through these events.
To compute a nearest-rank percentile, sort n observed durations and take the value at one-based position ceil(p × n), with p greater than zero and at most one. With 20 observations, p95 uses position 19 and p99 uses position 20. A sample containing nineteen 40 ms requests and one 400 ms request has mean 58 ms, p95 40 ms and p99 400 ms, so a p95-only dashboard would miss the slow request entirely. Other tools interpolate between positions, so compare definitions as well as labels.
The measurement population affects the result. A dashboard that records only successful completions may omit requests that time out, making an overloaded service appear faster as more slow requests fail. Keep the time window, observation count and error rate with latency summaries. Twenty observations illustrate the calculation but cannot provide a stable estimate of a production p99, which needs thousands of requests per window.
Component percentiles also cannot be added to obtain an end-to-end percentile. A database’s slowest requests may be different from a recommendation service’s slowest requests, and their correlation changes the total. Measure full request durations or inspect matched traces for individual requests. Summing durations is valid within one trace because the intervals belong to the same request and their dependency order is known.
What stays true
A response is ready only when every required result on its path has finished, so its latency is the finish time of the last required result.
In the example, the first three intervals total 55 ms. With independent branches, the profile read ends at 80 ms and recommendations at 115 ms; replying before 115 ms would omit a required result. In a larger dependency graph, each operation starts at the latest finish time among its prerequisites.
Reduce the interval that still limits the response
In the parallel default, reducing the 25 ms profile read to 5 ms saves no response time because recommendations still finish at 115 ms. Reducing recommendations from 60 to 30 ms lowers the request to 85 ms, saving 30 ms. The slower branch can change after an optimization, so recompute the path.
Monitoring systems rarely sort every observation. They usually aggregate durations into histogram buckets or bounded summaries and estimate percentiles from those, which introduces approximation error that depends on the bucket boundaries. A reported p99 therefore needs its aggregation method and window as well as its unit.
Each latency fix has a cost elsewhere. Parallel requests reduce elapsed time at the cost of concurrent resource use. Caching reduces a dependency’s work but introduces freshness and miss behavior. Moving data to another region cuts round trips for nearby readers while adding replication and consistency work. Removing an unnecessary dependency reduces both latency and failure exposure, while returning optional data later changes the response contract.
Timing mistakes that survive plausible arithmetic
Adding parallel branches as though they ran serially.
- Where it breaks
- The default independent calls are reported as 140 ms instead of 115 ms.
- The fix
- Draw their shared start and use the finish time of the slower required branch.
Reading microseconds as milliseconds.
- Where it breaks
- A 500 microsecond operation is entered as 500 ms instead of 0.5 ms, inflating its duration by 1,000.
- The fix
- Convert to one unit before calculating.
Expecting more bandwidth to fix a chatty cross-region page.
- Where it breaks
- Twenty sequential calls to a region 150 ms away still take about 3 seconds after the link is upgraded from 1 to 10 Gbps.
- The fix
- Batch the calls, run independent ones in parallel or move the data closer to the caller.
Using the mean as the experience of a slow user.
- Where it breaks
- The sample mean is 58 ms while the slowest of the 20 requests takes 400 ms.
- The fix
- Inspect percentiles, request counts and failed requests over the same window.
Optimizing a branch that finishes before the critical branch.
- Where it breaks
- Reducing the profile read from 25 ms to 5 ms still gives 115 ms in parallel mode.
- The fix
- Measure the critical path and confirm the end-to-end result after an optimization.
Connect timing to workload and request behavior
Related topics
- How the web works →
Expand the network interval into DNS, connection setup and response delivery.
- Back-of-envelope estimates →
Estimate request rates and bandwidth; elapsed time alone does not give serving capacity.
- Tail latency →
Why a request that waits for many servers sees their slowest responses, and how hedging helps.
Further reading
Books1
Links3
- Colin Scott: Latency numbers every programmer should know An interactive version of the table Jeff Dean popularized, with a slider showing how each row changed with hardware over the years. colin-scott.github.io (opens in a new tab)
- Google SRE: Monitoring Distributed Systems Latency distributions and distinguishing successful request latency from failed requests. sre.google (opens in a new tab)
- Latency numbers every programmer should know The classic table, based on Jeff Dean's talks. gist.github.com (opens in a new tab)