Question 872

Draft and Verify

In speculative decoding, a small draft model proposes tokens and a large target model verifies them in one parallel forward pass. What determines whether a drafted token is accepted while guaranteeing the final output distribution exactly matches sampling from the target model alone?

A

Accept if the draft model's softmax confidence exceeds a fixed threshold

B

Accept if the draft model's raw logit exceeds a temperature-scaled cutoff value

C

Modified rejection sampling: accept with probability min(1, p_target(x)/p_draft(x)), otherwise resample from an adjusted residual distribution

D

Accept only if the draft model's greedy argmax token equals the target model's greedy argmax token

E

Accept every proposed token up to the first point where beam search hypotheses diverge