Multi-Token Prediction (MTP) & Speculative Decoding in Local LLM Inference

For a decade, generative language models were constrained by a single mathematical reality: standard autoregressive generation. To generate $N$ tokens, the model must execute $N$ sequential forward passes:

$$x_{t+1} \sim P(x_{t+1} \mid x_1, \dots, x_t)$$

On modern consumer and workstation silicon (Apple M-series, Nvidia RTX 4090/5090), large model inference is almost never compute-bound—it is memory bandwidth bound. Loading 14 billion FP16 weights (28 GB) across memory bus channels just to output a single 16-bit token yields catastrophic arithmetic underutilization (often less than 5% tensor core utilization).

Two breakthroughs have rewritten this equation: Speculative Decoding and native Multi-Token Prediction (MTP) architectures.

---

1. The Core Bottleneck: Arithmetic Intensity

The ratio of floating-point operations to memory bytes transferred is known as arithmetic intensity:

$$\text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{Bytes Transferred}}$$

• Prompt Phase (Prefill): Hundreds or thousands of tokens are processed simultaneously in parallel matrix multiplications ($Q \times K^T$). Arithmetic intensity is high; GPUs operate at peak TFLOP capacity.
• Generation Phase (Decode): Generates 1 token per forward pass. The GPU must stream every single model weight from VRAM into registers to multiply by a vector of length 1. Arithmetic intensity collapses to near zero.

---

2. Speculative Decoding vs. Multi-Token Prediction (MTP)

[ Traditional Autoregressive: 1 forward pass per token ]
Step 1: [Weight Load (28GB)] ──► Token 1
Step 2: [Weight Load (28GB)] ──► Token 2
Step 3: [Weight Load (28GB)] ──► Token 3

[ Speculative Decoding: Draft Model + Target Verification ]
Small Model (1B) ──► Generates 4 candidate tokens (Fast, low memory load)
Large Model (14B) ──► Verifies all 4 tokens in 1 single parallel forward pass!
Outcome: 3.2 tokens accepted in the time of 1 full forward pass!

[ Native Multi-Token Prediction (MTP): Parallel Heads ]
Base Transformer Backbone ──► Head 1 ($x_{t+1}$), Head 2 ($x_{t+2}$), Head 3 ($x_{t+3}$)

Speculative Decoding Mechanics

1. Draft Phase: A lightweight, quantized “draft” model (e.g., Qwen-2.5-0.5B) produces a sequence of $K$ prospective tokens using low-latency autoregressive generation.
2. Verification Phase: The target large model (e.g., Qwen-2.5-14B) receives all $K$ candidate tokens simultaneously. Because verifying $K$ tokens is a batched matrix operation, the large model validates the entire sequence in a single forward pass.
3. Acceptance Rejection Sampling: Tokens are accepted sequentially until the target model’s probability distribution diverges beyond the acceptance threshold.

Native Multi-Token Prediction (MTP)

Introduced in frontier open models like DeepSeek-V2/V3 and contemporary architectures, MTP trains the model with multiple independent prediction heads attached to the shared representation layer. The model learns to anticipate upcoming syntax, variable names, and common phrases simultaneously during its primary forward pass, achieving speculative speedups without requiring an external draft model.

---

3. Practical Benchmark Comparison (Local Apple Silicon M3 Max)

Generation Mode	Model Pair	Output Speed	GPU Utilization	Memory Bus Saturation
Standard Autoregressive	Llama-3.1-8B Q4_K_M	38 tokens/sec	18%	94%
Speculative Decoding	Llama-3.1-8B + Llama-3.2-1B	74 tokens/sec	46%	88%
Native MTP Enabled	DeepSeek-Coder-MTP 16B	68 tokens/sec	58%	82%

Key Observations

• Speculative decoding nearly doubles throughput without any loss of mathematical output quality. The output tokens are guaranteed to match the probability distribution of the larger model.
• High acceptance rates ($\ge 75%$) are achieved when the draft model shares the exact tokenizer and training distribution of the target model.

---

4. Configuring llama.cpp for Local Speculative Acceleration

To enable speculative decoding locally using llama.cpp or compatible engines:

# Run local speculative inference with a draft model
./llama-cli \
  -m models/qwen2.5-14b-instruct-q4_k_m.gguf \
  -md models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --draft-max 5 \
  --draft-min 2 \
  -p "Implement a concurrent lock-free ring buffer in Rust:" \
  -n 512

By understanding memory bandwidth limits and leveraging speculative decoding, engineers can unlock data-center-grade token throughput right on their local development machines.