8 / 9 Inference Performance

The P99 Penalty

A serving system uses iteration-level (continuous) batching to maximize tokens/sec/GPU. As the average batch size grows, p50 per-token decode latency stays roughly flat, but p99 latency degrades disproportionately. What is the primary systems-level cause?

A

p99 degrades because KV-cache eviction forces recomputation for the last request admitted to the batch

B

p99 degrades because larger batches require a synchronous all-reduce across data-parallel replicas

C

Each in-flight request's next-token latency depends on whatever else shares that iteration's batch, so newly arriving prefill work causes head-of-line blocking that inflates the tail far more than the median

D

p99 degrades because sustained load throttles the GPU's boost clock

E

p99 degrades because floating-point accumulation error grows with batch size