Question 1138

Line Out the Door

Offered load on your inference cluster now exceeds what the GPUs can serve. The admission queue is unbounded, p99 time to first token has climbed past two minutes, and clients give up at 30s while the server keeps generating their responses. Which responses actually help?

A

Add replicas, since a sustained arrival rate above service capacity cannot be fixed by scheduling alone

B

Increase the maximum batch size past what the KV cache can hold and let the engine preempt sequences as needed

C

Bound the queue and reject with a 429 once it is full, so callers learn immediately instead of waiting on a backlog

D

Cancel in-flight generation when the client has disconnected or its deadline passed, freeing KV cache and decode slots

E

Raise the client timeout to 180 seconds so that fewer requests are abandoned before the server gets around to serving them