Speculative Decoding (cross-module)

Speculative Decoding is a lossless speedup for autoregressive generation: a lightweight draft model predicts $\gamma$ tokens, the target LLM verifies them in parallel — one forward pass computes logits at $\gamma+1$ positions — accepting the matching prefix and correcting mismatches via acceptance-rejection sampling. Speedup $E \approx \frac{1-\alpha^{\gamma}}{1-\alpha}$ ($\alpha$ = acceptance rate, $\gamma$ = draft length); EAGLE/Medusa instead add tree-structured heads to the target model to predict multiple future tokens, needing no separate draft model.

low-latency online inference (token-by-token autoregression is the main latency bottleneck); the largest gains come on memory-bandwidth-bound servers (larger $\gamma$ helps more); Medusa/EAGLE need no draft model, are simple to deploy, and are the mainstream single-model choice.

each autoregressive step emits one token while reading the full weight matrix — memory-bound and extremely low throughput. Parallel verification lifts effective output to about $\frac{1}{1-\alpha}$ tokens per step — e.g., $\alpha=0.8$, $\gamma=4$ gives $E \approx 2.95\times$; and acceptance-rejection sampling keeps the output distribution identical to the original model, making the speedup strictly lossless.

Test Your Knowledge: Practice Questions for "Speculative Decoding (cross-module)"

Single choice pitfall questions with instant feedback and mistake tracking.