Keep the GPU Fed

Compared to static batching, where a fixed group of requests runs together until every sequence in the batch finishes generating, why does continuous (in-flight) batching raise decode throughput?

It slots newly arrived requests into the batch as soon as any sequence finishes, avoiding idle time spent waiting on the longest sequence

It compresses model weights so fewer bytes are reread per decode step

It increases the numerical precision used for the KV cache

It skips attention computation entirely for shorter sequences

It pads every request to a fixed maximum length so the batch only needs one dispatch