Split Personality

A disaggregated LLM serving architecture (e.g., DistServe) runs prefill and decode on separate GPU pools instead of colocating them inside one continuous-batching engine. What is the primary reason for this separation, and what does it require to work?

Decode needs exclusive access to L2 cache banks that prefill's attention kernel evicts, so disaggregation exists to prevent cache pollution between kernel types rather than for compute/memory-bandwidth reasons.

Disaggregation exists mainly so decode instances can autoscale independently in Kubernetes, something impossible if prefill and decode share one process's batching loop.

Prefill is compute-bound (one large batched forward pass over the full prompt) while decode is memory-bandwidth-bound (many small batched passes streaming an ever-growing KV cache); colocating them lets prefill iterations stall decode's per-token latency, so each phase is put on GPUs with independently tuned parallelism (more tensor parallelism for prefill, more pipeline stages for decode), at the cost of transferring the KV cache over the network from prefill to decode workers.

Prefill instances hold the full fp16 weights while decode instances only need INT4-quantized weights, so the split exists purely to save GPU memory on the decode side.

Prefill and decode use different tokenizer vocabularies (byte-level vs subword), so the split is required for correctness, not performance.