Orca: A Distributed Serving System for Transformer-Based Generative Models(opens in a new tab)
Iteration-level scheduling, now usually called continuous batching: the server re-forms the batch after every decoding step, so finished requests leave and new ones join without waiting for the whole batch. Selective batching makes this work for attention.