Speculative Speedup, Actually

A team runs speculative decoding with a draft model that proposes $\gamma=4$ tokens per cycle at cost ratio $c=0.1$ relative to the target model (draft forward pass costs 10% of a target forward pass), and rejection-sampling verification gives an independent per-token acceptance rate $\alpha=0.8$ . Using the standard expected-speedup model $S = \dfrac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(1 + \gamma c)}$ , what's the approximate speedup over plain autoregressive decoding?

$\sim 4.8\times$ , since $\gamma+1$ tokens are proposed per cycle at a $0.8$ acceptance rate

$\sim 5.0\times$ , since $\dfrac{1}{1-\alpha}$ alone gives the ceiling on tokens per cycle

$\sim 3.4\times$ , since that's just the expected number of tokens accepted per cycle before verification overhead

$\sim 2.4\times$ , once the extra draft-model forward passes are amortized into the denominator

$\sim 1.0\times$ , because the draft model's added compute roughly cancels out the tokens it saves