Batch Size Tug-of-War

A team serving an LLM API is deciding how aggressively to batch concurrent user requests together on the GPU. What tradeoff does increasing the batch size at inference time create?

Batch size affects only training stability and has no measurable effect on inference performance

Higher aggregate throughput (tokens/sec across all users) at the cost of higher per-request latency, since each request waits longer in a bigger batch

Higher throughput with no latency cost, since GPUs process batched requests in parallel with zero added wait

Lower throughput but lower latency, since smaller per-request compute chunks finish faster