Inference 0/0

Inference

Batching and serving

Prefill and decode, time to first token and inter-token latency, the roofline, continuous batching and disaggregated serving. Serving engines such as vLLM and SGLang are built around these tradeoffs.

0 / 9 solved

Reading

WhiteBox lessons1

Papers and essays3

  1. Orca: A Distributed Serving System for Transformer-Based Generative Models(opens in a new tab)

    Yu et al. 2022 OSDI 2022 60 min

    Iteration-level scheduling, now usually called continuous batching: the server re-forms the batch after every decoding step, so finished requests leave and new ones join without waiting for the whole batch. Selective batching makes this work for attention.

  2. SGLang: Efficient Execution of Structured Language Model Programs(opens in a new tab)

    Zheng et al. 2024 NeurIPS 2024 60 min

    SGLang: a language for programs that make many LLM calls, and a runtime whose RadixAttention keeps KV caches in a radix tree so calls that share a prompt prefix reuse it, plus compressed state machines for faster structured output.

  3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving(opens in a new tab)

    Zhong et al. 2024 OSDI 2024 60 min

    Disaggregated serving: run prefill and decoding on separate GPUs and choose resources and parallelism for each phase against its own latency target, time to first token or time per output token.

Videos2

LLM Inference and GPU Performance Various speakers 2 videos

Links1