Question 1136

Stuck Behind the Long One

A serving stack forms a batch of 16 requests, runs them together, and only starts the next batch once every request in the current one has finished. Short requests that generate 20 tokens sit idle until a 2000-token request completes. What change directly fixes this?

A

Turn off batching and serve each request on its own pass so that no request ever waits behind another

B

Switch to continuous batching, where the scheduler retires finished sequences and admits queued ones after each decode step

C

Sort the pending queue by prompt length so that requests with similar prompt sizes land in the same batch

D

Raise the static batch size from 16 to 64 so many more requests share every pass through the model weights