Goodput, Not Just Grunt

A continuous-batching (iteration-level scheduling) engine like Orca/vLLM decides at every iteration whether to admit a new request's prefill into a batch already running decode steps. Naively admitting a large prefill causes a 'generation stall' that can blow the time-per-output-token (TPOT/TBT) SLO for in-flight decodes, which is why Sarathi-Serve instead chunks prefill and interleaves it with ongoing decodes (stall-free batching). Given this, which statement correctly distinguishes 'goodput' from raw throughput and explains why cranking up batch size does not raise goodput monotonically?

Goodput only counts requests served via speculative decoding, so bigger batches lower goodput purely by lowering the speculative-decode acceptance rate, unrelated to TBT.

Goodput is a compile-time property of the model's quantization scheme (INT8 vs fp16), independent of the scheduler, so batch size cannot affect it.

Goodput measures the request rate a system sustains while meeting SLOs (e.g., TTFT and TPOT/TBT percentile targets), not just tokens/sec; larger batches raise raw throughput by amortizing memory-bandwidth cost per token, but past some point each iteration's forward pass takes longer (more KV cache read, bigger matmuls, co-admitted prefill chunks), so TBT tail latency grows, requests start missing SLO, and those requests still count toward raw throughput but not toward goodput.

Goodput and throughput are the same metric at different time granularities (goodput is throughput averaged over a minute), so batch size affects both identically and no such tradeoff exists.

Bigger batches always raise both throughput and goodput identically because GPUs are throughput-oriented; production systems cap batch size only to leave VRAM headroom for future KV caches, not for latency reasons.