Question 1137

Slow Start, Smooth Finish

After a product change, users paste long documents into the chat. Median time to first token rose from 0.4s to 2.5s while inter-token latency once streaming begins is unchanged. Requests share a large identical system prompt and document header. What explains this and what helps most?

A

The concurrent sequence limit is set too low, so raising the batch cap will bring the first token back under a second

B

Prefill must run over every prompt token before the first output token, and caching the KV blocks of the shared prefix removes that repeated work

C

The KV cache is full, so active sequences are being preempted and then recomputed from scratch, and that recomputation is what delays the first token

D

Decode is bound by memory bandwidth, so moving to a GPU with more bandwidth is the direct fix for a slower first token