What "speculative decoding" actually means

Every time you ask an LLM to write something, it produces one token at a time. The model reads what it has already written, runs its full transformer forward pass, and picks the next word. That repetition is slow — it's exactly like writing a document by hand, waiting for every character to dry before typing the next one.

Speculative decoding changes that workflow. A smaller, much faster draft model (often called a "boombot" after the original paper [1]) generates several tokens ahead in parallel. A larger target model then verifies all those draft tokens at once. If they match what the big model would have produced, they get accepted. Where they disagree, the draft stops and the target model takes over from that point.

The result: fewer transformer passes for the same output. When acceptance rates are high — which they often are in practice — you can get 2x to 4x speedups without any quality loss. The original text stays identical; only the generation process changes.

The mechanics (no math required)

Here's what happens step by step when a prompt comes in:

  1. Draft phase. A small model (say, a 7B parameter model with no cross-attention layers) autoregressively generates N candidate tokens from the prompt. These are speculative guesses about what the target model would produce next.
  2. Verify phase. The large target model receives all N draft tokens plus the original prompt in a single forward pass. It checks each draft token against its own predicted distribution.
  3. Accept or reject. For every draft token that lands in the top choice of the target model's output, the system accepts it. For the first mismatch, it replaces that token with what the target model predicts and may even accept subsequent tokens drawn from the correct residual distribution [1].

The key insight: you don't need a perfect draft model. You just need a good enough one that guesses correctly most of the time. Acceptance rates between 60–80% are typical for draft models roughly one order of magnitude smaller than the target [2].

Why "smaller doesn't have to mean worse"

Early speculation assumed the draft needed to be a miniature of the target — same architecture, fewer layers. That was wasteful. Recent work has shown that draft models don't inherit from the target at all; they just need to understand the same data distribution, which any good model does [3]. This is important for LLM Ops hitl workflows where you might want a local 1B-7B boombot to speculate for a server-side larger model without deploying a second instance of yourself.

The practical implication: if you already have a fast smaller model on your hardware (and Ollama makes this trivial), you can use it as a speculative draft with essentially zero additional setup beyond configuration changes in vLLM or TensorRT-LLM [4].

What's changed in 2025–2026

Three converging advances made speculative decoding practical for production:

  • Diverse architectures work. Draft and target models no longer need to be related. A Qwen-style model can draft for a Llama-family model.
  • Open-source implementations landed in vLLM and TensorRT-LLM with native support, meaning you don't roll your own inference loop.
  • Multi-draft schedules allow adaptive token counts based on the current stage of generation (early tokens get more speculative freedom; late tokens get fewer but targeted guesses).

As Google Research summarizes it, speculative decoding has become an effective technique for faster and cheaper LLM inference without compromising quality. That last phrase matters — the generated text is bit-for-bit identical when all tokens are accepted.

Where speculative decoding fits in your stack

For teams running local inference under budget constraints, speculative decoding is one of the few optimizations that delivers measurable speedup without changing prompts or fine-tuning data. It pairs naturally with HITL workflows because the human verification step — checking output quality before any outbound action — remains unchanged regardless of how many tokens the system generated per second.

The draft model itself runs within your infrastructure (local, air-gapped if you choose), so there are no new external API dependencies to manage. It extends an existing pipeline's capacity at essentially zero incremental cost.

In LLM Ops hitl terms: faster inference → lower cost per query → more queries you can afford to run through a dry-run build before the Board approves anything live. That's compounding value.

What to watch next in speculative decoding research

A few directions are active now that deserve attention:

  • Cross-modal drafting. Speculation applied between text and vision/text encoders for multimodal generation [5]. Could a text draft model help accelerate image captioning pipelines?
  • Asynchronous multi-device speculation. Distributing the verify phase across multiple accelerators so the bottleneck stays on the draft side, not the verifier [6].
  • Draft quality estimation. Methods to predict acceptance rates *before* running verification, so you can schedule compute smarter.

These won't arrive overnight, but speculative decoding as a technique is mature enough that teams are now benchmarking it in production rather than just reading papers about it.

In practice: what this means for your workflow

If you run Ollama with a larger model and have a smaller model on the same GPU:

```bash

Draft with a small, fast model; verify with a large one

ollama run qwen3:1b # draft — runs at ~10x throughput

ollama run qwen3:32b # target verifier — runs as usual

```

Configure vLLM to treat the first as a speculative draft and the second as the target, and you're done. No data retraining. No new infrastructure. Your HITL review remains exactly where it should be — focused on output quality, not generation speed.

---

_This article is educational material from MeltingFace's Presence living lab, documenting techniques we evaluate for agentic-first operations. Nothing here constitutes a product announcement or service offering._

_MeltingFace is an LLM Ops research consultancy. Presence is our HITL living lab — not a chatbot vendor._

_See our work on local inference workflows or our engineering philosophy for related reading._

---

References

  1. Leviathan, et al., "Fast Inference from Transformers via Speculative Decoding," ICML 2023. <https://arxiv.org/abs/2211.17192>
  2. Original paper analysis: acceptance rates of 60–80% for models ~10× smaller in parameter count than target.
  3. "Looking back at speculative decoding," Google Research blog (2024). <https://research.google/blog/looking-back-at-speculative-decoding/>
  4. vLLM speculative execution docs; TensorRT-LLM multi-token generation documentation.
  5. "Speculative Decoding for Multimodal Models: A Survey," Preprints.org, 2026. <https://www.preprints.org/manuscript/202603.2344>
  6. AMD/ISCAS 2025 work on asynchronous multi-device speculative decoding.

MeltingFace