Speculative Decoding with Draft Models

Speculative Decoding & Draft-Model Inference Acceleration delivers mathematically lossless 2-3x throughput acceleration for autoregressive LLM decoding by exploiting compute-bound parallel verification against memory-bound single-token generation bottlenecks; the 3-step pipeline comprises: 1) Speculative Proposal: a lightweight, ultra-fast draft model (e.g. 1B model) autoregressively drafts $K$ candidate tokens sequentially; 2) Parallel Verification: the large target model (e.g. 70B model) executes a single parallel forward pass across all $K$ proposed tokens simultaneously, computing target probability distributions in one matrix operation; 3) Rejection Sampling: sequentially evaluating each token under acceptance threshold $\alpha_i = \min(1, \frac{P_{\text{target}}(x_i)}{P_{\text{draft}}(x_i)})$; if a token is rejected, the target model samples a replacement token and discards remaining drafts, guaranteeing exact mathematical equivalence to native target generation.

Low-latency conversational streaming, real-time code auto-complete, and cloud LLM serving speedups.

Single-token decoding is memory-bandwidth bound where reading 70GB of weights produces only 1 token; speculative decoding validates multiple tokens per memory load, cutting TPOT latency in half.

Derive the mathematical proof showing why Speculative Decoding's rejection sampling criterion produces zero distribution divergence from $P_{\text{target}}(x)$?

Derive the theoretical speedup multiplier equation as a function of draft length $K$, acceptance rate $\alpha$, and relative draft-to-target cost?

Contrast Draft-Free speculative methods: Medusa's multi-head classification heads vs Eagle's hidden-state contextual autoregression?

Explain Prompt Lookup Decoding which extracts candidate token n-grams directly from the input prompt without needing draft models?

How does vLLM orchestrate target and draft model memory co-location and parallel tensor scheduling during multi-tenant serving?