Question 735

Reading TTFT and ITL Off One Request

A benchmark run against one serving replica reports these numbers for a single request:

measurement value
input prompt 3,000 tokens
output generated 401 tokens
time to first token 0.40 s
end-to-end request latency 8.40 s

GenAI-Perf defines inter-token latency as ITL = (e2e_latency - TTFT) / (total_output_tokens - 1). What is this request's ITL, and if the prompt doubles to 6,000 tokens with the same output length, which of the two numbers moves first?

A

20.0 ms per token, and doubling the prompt doubles ITL too, because every decode step re-runs the prompt through the model.

B

20.0 ms per token. Doubling the prompt roughly doubles prefill work, so TTFT is what grows; ITL is set by the per-step decode loop and stays near 20 ms.

C

19.95 ms per token, spreading the 8.00 s of generation across all 401 output tokens.

D

20.9 ms per token, dividing the full 8.40 s end-to-end latency by the 401 output tokens.