Waiting On Request B
A server holds four decode slots. It admits a batch, runs one decode iteration per step for every sequence in the batch, and does not admit anything new until the whole batch has finished. Four requests arrive together:
| request | output tokens before its stop token |
|---|---|
| A | 12 |
| B | 400 |
| C | 20 |
| D | 15 |
Eight more requests are queued behind them.
Count a slot-iteration as one decode slot occupied for one iteration. How many of the slot-iterations this batch consumes are spent on sequences that have already emitted their stop token?
Sign in to answer questions and track your progress
Sign In