The Stream Stutters

Your streaming chat product runs on vLLM with chunked prefill enabled. Users complain that generation "stutters": tokens arrive smoothly for a second, then pause for several hundred milliseconds, then resume. Time to first token is excellent. The pauses correlate with other users pasting long documents.

llm = LLM(
    model=MODEL,
    gpu_memory_utilization=0.90,
    max_num_seqs=256,
    max_num_batched_tokens=32768,   # tokens scheduled per engine step
)

Which change addresses the complaint, and what does it cost you?

Raise gpu_memory_utilization to 0.97. More KV cache means fewer preemptions, and preemption is what causes the pauses. The cost is less headroom for activation peaks.

Lower max_num_batched_tokens to around 2,048. Fewer prefill tokens ride along in each step, so inter-token latency smooths out. The cost is time to first token on long prompts, which now clear in many more chunks.

Raise max_num_batched_tokens to 65,536 so a long prompt clears its prefill in one step and stops interfering with decodes. The cost is a larger activation footprint per step.

Lower max_num_seqs to 32. Fewer concurrent sequences means fewer decode steps competing for the GPU. The cost is aggregate throughput, since the batch is smaller.