The Speculation Depth Trade-Off: Pushing EAGLE MTP to Three Steps on Blackwell
A Single Benchmark That Reveals the Diminishing Returns of Deeper Speculation
In the high-stakes world of large language model serving, every token per second counts. When deploying the 122-billion-parameter Qwen3.5-122B-A10B-FP8 model across four NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant in this opencode session found itself navigating a classic engineering trade-off: how many speculative decoding steps should you use? More speculation means more tokens generated per forward pass, but it also consumes more KV cache per request, reducing the number of concurrent requests the system can handle. Message [msg 6508] captures the moment the assistant pushed this knob to its third position — speculative_num_steps=3 — and measured the result.
The Chain of Discovery
To understand message [msg 6508], we must trace the thread that led to it. The assistant had been systematically optimizing the Qwen3.5-122B-A10B-FP8 deployment, a Mixture-of-Experts model with 256 experts and a per-expert intermediate size of just 256 — tiny GEMM operations that make the MoE kernel configuration largely irrelevant. After ruling out kernel autotuning as a meaningful lever (see [msg 6494]), the assistant turned to speculative decoding, specifically the EAGLE-style Multi-Token Prediction (MTP) that SGLang supports.
The baseline configuration used speculative_num_steps=1 (producing 2 draft tokens per step: 1 original + 1 speculated). Benchmarking at [msg 6491] showed a respectable 122.9 tok/s aggregate single-request throughput. The assistant then increased speculative_num_steps to 2 at [msg 6502], yielding 182.7 tok/s — a stunning 49% improvement over baseline. The single-request per-req throughput jumped from 124.3 to 185.7 tok/s, a +51% gain. This was a massive win, but it came with a cost: max_running_requests dropped from 26 to 21, and max_total_num_tokens fell from 455,357 to 377,207. Each speculation step consumes KV cache headroom.
The natural question: would a third step continue the trend?
Message 6508: The Third Step
Message [msg 6508] is deceptively brief. The assistant writes:
speculative_num_draft_tokens=4(3+1),max_running_requests=17,max_total_num_tokens=295931. More draft tokens = less KV cache. Let's see if the extra speculation step is worth it.
Then executes the benchmark. The results:
C | Agg tok/s | Per-req tok/s | Tokens | Wall(s) | Ok
-----------------------------------------------------------------
1 | 229.1 | 234.0 | 4000 | 17.5 | 4
4 | 705.0 | 199.7 | 7369 | 10.5 | 8
16 | 1733.0 | 119.9 | 29651 | 17.1 | 32
32 | 1978.5 | 119.8 | 51000 | 25.8 | 64
64 | 1913.8 | 119.5 | 98000 | 51.2 | 128The single-request throughput of 234.0 tok/s represents a +90% improvement over the baseline of 124.3 tok/s. Compared to the steps=2 configuration at 185.7 tok/s, this is a further +26% gain. The aggregate throughput at 64 concurrent requests reaches 1,914 tok/s, up from 1,844 tok/s with steps=2 and 1,566 tok/s at baseline.
But the diminishing returns are already visible. The jump from steps=1 to steps=2 delivered +51% in single-request throughput; the jump from steps=2 to steps=3 delivered only +26%. Meanwhile, the KV cache cost continued to mount: max_running_requests dropped from 21 to 17, and max_total_num_tokens fell from 377,207 to 295,931 — a 22% reduction in total token capacity for a 26% gain in single-request throughput. The trade-off is tightening.
The Implicit Reasoning
Though the assistant does not include an explicit chain-of-thought block in this message, the reasoning is encoded in its structure. The assistant opens by stating the configuration parameters and their implications — "More draft tokens = less KV cache" — acknowledging the cost before even measuring the benefit. This reveals a mental model of the system as a constrained optimization problem: maximize throughput subject to KV cache capacity.
The decision to test steps=3 was a direct extrapolation from the success of steps=2. But the assistant also understood that the marginal benefit would likely shrink. In speculative decoding, the acceptance rate of draft tokens decreases with each additional speculation step — the model's predictions become less accurate the further ahead they try to look. The first speculated token (step 1) has the highest acceptance probability; the second (step 2) is lower; the third (step 3) is lower still. At some point, the cost of storing the extra draft KV cache entries exceeds the benefit of the occasional accepted token.
The assistant's choice of benchmark parameters — temperature=0, ignore_eos=True — is worth noting. As the assistant itself acknowledged in [msg 6502], this configuration "artificially inflates acceptance rates (repetitive output is highly predictable)." With temperature=0, the model always selects the most likely token, and with ignore_eos=True, it never terminates early, producing a steady stream of highly deterministic tokens. This is the best-case scenario for speculative decoding. Real-world performance, especially for agentic coding workloads with diverse output distributions, would likely be lower.
What the Results Actually Tell Us
The benchmark at [msg 6508] provides several insights:
Single-request throughput scales with speculation depth, but with diminishing returns. The progression from 124 → 186 → 234 tok/s shows that each additional step adds value, but the increment shrinks. The marginal gain of +48 tok/s from step 2 to step 3 is still meaningful, but it comes at a disproportionate cost in KV cache capacity.
Per-request throughput at high concurrency converges. At 64 concurrent requests, per-req tok/s is nearly identical across all three configurations: ~65 (steps=1), ~99 (steps=2), ~120 (steps=3). The gap narrows because at high concurrency, the system is bottlenecked by compute rather than speculation efficiency. The speculation depth primarily helps latency-sensitive single-request scenarios.
The KV cache trade-off is real but manageable. The system still achieves 1,914 tok/s aggregate at 64 concurrent requests with steps=3, compared to 1,566 tok/s at baseline. Even with reduced max_running_requests, the overall throughput is higher because each individual request completes faster, freeing up capacity sooner.
The Broader Context
This message sits within a larger arc of deployment optimization spanning segments 37 through 42 of the opencode session. The assistant had previously struggled with EAGLE-3 speculation on this model (see segment 37), finding that the baseline strictly outperformed EAGLE-3 at all concurrency levels. The pivot to SGLang's built-in MTP speculation (the --speculative-num-steps and --speculative-eagle-topk flags) proved far more successful.
The deployment target — Qwen3.5-122B-A10B-FP8 on 4× RTX PRO 6000 Blackwell GPUs — is a cutting-edge configuration. The FP8 quantization, Mixture-of-Experts architecture with 256 experts, and the Blackwell SM120 compute architecture all interact in complex ways. The assistant had already spent significant effort building SGLang from source, applying SM120 patches, fixing FP8 KV cache accuracy, and configuring NCCL for tensor parallelism. The speculation tuning in [msg 6508] represents the final layer of optimization on top of a carefully constructed stack.
Conclusion
Message [msg 6508] is a masterclass in systematic performance tuning. The assistant identifies a lever (speculation depth), measures its effect, acknowledges the trade-off, and tests the next increment. The results — 234 tok/s single-request, 1,914 tok/s aggregate — represent a 90% improvement over the starting point, achieved entirely through configuration changes without any code modifications.
But the message also reveals the limits of this approach. The diminishing returns from steps=2 to steps=3, combined with the acknowledged artificiality of the benchmark conditions, suggest that further increases to steps=4 or beyond would yield minimal benefit. The assistant implicitly understands this — the message ends with the benchmark results and no immediate plan to push further. The next step would likely be to validate these results under more realistic conditions, or to explore other optimization axes entirely.
In a single, compact message, the assistant demonstrates the essence of engineering: measure, trade off, decide, and move on.