A new technical post details how speculative decoding speeds LLM inference without sacrificing accuracy

Researchers have published the third installment in a series on the co-design of AI models, this time focusing on speculative decoding, a technique that accelerates the auto-regressive decoding step by predicting multiple tokens per iteration. A small draft model proposes several plausible tokens, which are then verified in a single forward pass through the larger target model. The approach reduces the total number of decoding iterations and raises the arithmetic intensity of the target model without requiring higher concurrency.

How it works in practice

The target model accepts the proposed tokens sequentially until it encounters the first mismatch; the next prediction cycle then resumes from that point. Because only tokens approved by the target model are retained, speculative decoding produces an output sequence identical to standard decoding unless the acceptance criteria are deliberately relaxed. Draft length (D) denotes the number of tokens proposed per target iteration, while acceptance length (AL) is the number of tokens actually accepted; AL ranges from 1 to (1 + D), since the target model can always emit one additional ground-truth token beyond the accepted draft tokens.

The formula behind the speedup

Speedup is measured as the ratio of the time the target model takes to generate AL tokens sequentially to the time required to verify D tokens in parallel plus the draft-generation latency overhead. Ignoring draft-model latency, speculation yields a speedup when the batch verification time multiplied by (1 + D) is less than AL. During verification, compute scales with (1 + D) while memory access remains constant, so the objective is to increase D until verification time stops growing — that is, until verification transitions from memory-bound to compute-bound. The optimal D depends on batch size B and shifts along the Pareto frontier.

GEMM-M grows, a small batch suffices

With speculation, the GEMM-M dimension of every linear layer in the target model expands from M to M × (1 + D). The data show that higher draft lengths let GEMMs reach peak performance at lower effective batch sizes. Notably, with D = 7 only one-eighth of the batch size is needed to become compute-bound compared with D = 0. As mixture-of-experts (MoE) models grow sparser and long-context workloads increase pressure on KV-cache capacity, effective concurrency per expert declines, making larger draft lengths more attractive.

Five guidelines for practical selection

The full post lays out five guidelines for choosing draft length and draft mechanism along the Pareto frontier, accounting for the tension between throughput and interactivity. Code for the series is available in the project's open repository; earlier parts covered hardware-friendly dense LLM design and fast attention for long context.