Fast Inference from Transformers via Speculative Decoding(opens in a new tab)
Speculative decoding: a small draft model proposes several tokens, the large model checks them all in one parallel pass, and a modified rejection-sampling rule keeps the output distribution identical to the large model's.