Inference 0/0

Inference

Speculative decoding

A small draft model proposes several tokens and the large model checks them all in one forward pass, keeping its exact output distribution. It speeds up decoding whenever most drafted tokens are accepted.

Reading

Papers and essays1

  1. Fast Inference from Transformers via Speculative Decoding(opens in a new tab)

    Leviathan, Kalman and Matias 2023 ICML 2023 50 min

    Speculative decoding: a small draft model proposes several tokens, the large model checks them all in one parallel pass, and a modified rejection-sampling rule keeps the output distribution identical to the large model's.

Videos1

LLM Inference and GPU Performance Various speakers 1 video