Prefill/Decode Split

Prefill (compute-bound, processes the whole prompt at once) and decode (memory-bandwidth-bound, one token per step) have very different resource profiles but are traditionally batched together on the same GPUs. Contrast chunked prefill and prefill/decode disaggregation, and the tradeoff each makes.

Chunked prefill moves the prefill computation to CPU while decode stays on GPU, trading network latency for freed GPU compute; disaggregation instead keeps both phases on GPU but forces them into strictly alternating time slices on one shared cluster.

Both techniques avoid any KV-cache transfer by keeping prefill and decode for a given request pinned to the same GPU at all times; the only difference between them is the batch scheduling order used, FIFO versus priority.

Chunked prefill splits a long prompt's prefill into smaller pieces piggybacked into ongoing decode batches on the same GPUs, improving utilization and smoothing latency with no extra hardware, but prefill and decode still contend for the same compute; disaggregation instead runs prefill and decode on physically separate GPU pools connected by a KV-cache transfer step, removing that interference and letting each stage scale independently, at the cost of extra cross-node KV transfer and more hardware to orchestrate.

Chunked prefill and disaggregation are two names for the same technique: both split the prompt into fixed 128-token chunks and run each chunk on its own GPU, differing only in which open-source project implements them rather than in system design.

Chunked prefill always increases time-to-first-token by delaying prefill until a decode batch is full, and disaggregation always increases total GPU count with no throughput benefit, so both techniques are strictly latency-worse and are only ever used to cut cost.