The Speculation Depth Tuning: When Two Steps Beat One in Multi-Token Prediction

In the long arc of deploying and optimizing the Qwen3.5-122B-A10B model on a cluster of NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 6502] represents a quiet but pivotal moment. It is not a message of grand architectural decisions or dramatic bug fixes. It is a message about a single parameter change — increasing speculative_num_steps from 1 to 2 — and the careful measurement of its consequences. Yet within this seemingly narrow adjustment lies a rich story of hypothesis-driven optimization, resource trade-off analysis, and the kind of deep systems thinking that separates a working deployment from a truly performant one.

The Road to This Message

To understand why the assistant made this change, we must trace the path that led there. The preceding messages ([msg 6472] through [msg 6501]) document a failed attempt at MoE kernel autotuning. The assistant had spent considerable effort trying to improve throughput by tuning the fused MoE Triton kernel for the Blackwell architecture. It patched common_utils.py to fix a bug where the script's text_config redirect lost the model architecture information. It tried running the autotuner, only to find it timed out after 10 minutes for a single batch size. It then pivoted to copying the B200 configuration file as a starting point, reasoning that both B200 and RTX PRO 6000 are Blackwell-family GPUs (SM100 vs SM120) with similar microarchitectures.

The benchmark with the B200 config ([msg 6491]) revealed the uncomfortable truth: the numbers were identical to the default configuration within noise. The assistant correctly diagnosed why: "With E=256 experts but only N=256 per-expert intermediate size (very small matrices per expert), the GEMM is memory-bound not compute-bound. The kernel config doesn't matter much when the matrices are tiny." This is a crucial insight — the MoE kernel was not the bottleneck for this model architecture, and no amount of kernel tuning would move the needle.

This realization forced a strategic pivot. If the MoE kernel wasn't the bottleneck, what was? The assistant turned its attention to speculative decoding, specifically the Multi-Token Prediction (MTP) / EAGLE-style speculation that SGLang implements. The model was already running with --speculative-eagle-topk 1, which means it drafts a single candidate token per speculation step. The assistant hypothesized that increasing the speculation depth — the number of steps the draft model looks ahead — could improve throughput, provided the acceptance rate was decent.

What the Message Contains

The message opens with a concise set of observations extracted from the server logs:

- speculative_num_steps=2, speculative_num_draft_tokens=3 (auto-adjusted: steps+1) - max_running_requests=21 (dropped from 26 with steps=1 — more draft tokens = more KV cache overhead per request) - max_total_num_tokens=377207 (dropped from 455357)

These three lines reveal the assistant's analytical mindset. Before even looking at the benchmark numbers, it checks the server's resource parameters to understand the cost of the change. The max_running_requests dropping from 26 to 21 is a real capacity reduction — the server can now handle 5 fewer concurrent requests because each request consumes more KV cache memory for the additional draft tokens. The max_total_num_tokens similarly drops by about 17%.

Then comes the benchmark. The assistant runs the same bench_qwen.py script with concurrency levels 1, 4, 16, 32, and 64:

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      182.7 |          185.7 |     4000 |     21.9 |    4
   4 |      542.6 |          162.4 |     7217 |     13.3 |    8
  16 |     1530.1 |          107.5 |    29408 |     19.2 |   32
  32 |     1672.5 |           97.9 |    51314 |     30.7 |   64
  64 |     1844.0 |           99.1 |   108000 |     58.6 |  128

The Reasoning Behind the Change

The assistant's decision to increase speculative_num_steps was grounded in a clear understanding of how EAGLE-style speculation works in SGLang. With speculative_eagle_topk=1, the system drafts exactly one candidate token per step. The total number of draft tokens per forward pass is speculative_num_steps + 1 (one for the original token plus one speculated token per step). With steps=1, that means 2 tokens per step. With steps=2, that means 3 tokens per step.

The assistant's hypothesis was that the model's acceptance rate was high enough that the extra speculation step would produce useful tokens more often than it would waste computation on rejected drafts. This is a non-trivial assumption — deeper speculation means more tokens are generated per forward pass, but each rejected token represents wasted computation. The optimal speculation depth depends on the model's empirical acceptance rate, which varies by model, task, and even input distribution.

The assistant did not have a direct measurement of the acceptance rate. Instead, it relied on a heuristic: if the single-step speculation was already providing a significant speedup (the baseline without speculation was around 122 tok/s for single requests, and steps=1 gave 124 tok/s — barely any improvement at low concurrency), then deeper speculation might help. The reasoning was that the draft model was conservative enough that its predictions were accepted frequently, and adding another step would compound the benefit.

Analyzing the Results

The benchmark results are striking. At single-request concurrency (C=1), the throughput jumps from 124.3 tok/s (with steps=1, measured in [msg 6491]) to 185.7 tok/s — a 49% improvement. This is a dramatic gain from a single parameter change. At C=4, the improvement is from 118.9 tok/s to 162.4 tok/s (37%). At C=16, from 84.9 to 107.5 (27%). At C=32, from 62.5 to 97.9 (57%). At C=64, from 65.3 to 99.1 (52%).

The improvement is not uniform across concurrency levels, which is expected. At low concurrency, the speculation overhead (computing draft tokens) is a smaller fraction of total work because the GPU is underutilized. At high concurrency, the GPU is saturated, and the speculation overhead competes with serving more requests. Yet even at C=64, the improvement is substantial.

What's particularly interesting is the shape of the throughput curve. With steps=1, the aggregate throughput peaked at 1566 tok/s at C=64 and was already plateauing. With steps=2, it reaches 1844 tok/s at C=64 and is still climbing (the per-request throughput at C=64 is 99.1 tok/s, higher than at C=32's 97.9 tok/s, suggesting the plateau hasn't been reached yet). This indicates that the deeper speculation is more efficiently utilizing the GPU's compute capacity, allowing it to sustain higher throughput even under heavy load.

The Trade-Offs: Capacity vs. Throughput

The assistant's observations about max_running_requests and max_total_num_tokens reveal a sophisticated understanding of the resource trade-offs. The server has a fixed memory budget for KV cache. Each request with deeper speculation consumes more KV cache slots per step because the draft tokens need to be cached for verification. This reduces the maximum number of concurrent requests the server can handle.

The drop from 26 to 21 concurrent requests represents a 19% reduction in capacity. Yet the throughput at high concurrency improved by 18-57%. The trade-off is favorable: the server loses some peak concurrency but gains more throughput per request, and the aggregate throughput is higher at every measured concurrency level. For a production deployment serving many concurrent users, this is a net win — the server can handle the same workload with fewer resources, or serve more tokens with the same resources.

However, the assistant does not explicitly discuss whether the reduced max_running_requests could become a bottleneck under very high load. If the workload requires serving more than 21 concurrent long-context requests, the server would queue requests, potentially increasing tail latency. This is a subtle point that a less experienced engineer might miss, but the assistant flags the trade-off implicitly by reporting the numbers.

The Thinking Process Visible in This Message

The message reveals a clear experimental methodology. The assistant:

States the hypothesis implicitly: By changing speculative_num_steps from 1 to 2, it expects improved throughput if acceptance rate is decent.
Checks the configuration was applied correctly: It reads the journal logs to confirm the server picked up the new parameters.
Quantifies the resource cost: It notes the reduction in max_running_requests and max_total_num_tokens.
Runs a controlled benchmark: It uses the same benchmark script with the same concurrency levels as the previous test, ensuring apples-to-apples comparison.
Reports results without commentary: It lets the numbers speak for themselves, trusting that the reader (or the user) can interpret the improvement. This is a textbook example of how to evaluate a performance optimization: measure the baseline, change one variable, measure again, and report both the benefits and the costs.

Assumptions and Limitations

The assistant makes several assumptions that are worth examining:

The acceptance rate is high enough: This is the core assumption. If the draft model's predictions were frequently rejected, deeper speculation would waste compute. The results validate this assumption empirically, but the assistant never directly measures the acceptance rate.
The benchmark is representative: The bench_qwen.py script generates synthetic prompts with fixed output lengths. Real-world traffic patterns might have different acceptance rates, especially for tasks that require specific token patterns (code generation, structured outputs, etc.).
The KV cache overhead is acceptable: The assistant implicitly accepts the 19% reduction in concurrent capacity as a worthwhile trade-off. For workloads with very high concurrency requirements, this might not be the right choice.
No other parameters need adjustment: The assistant changes only speculative_num_steps without adjusting speculative_num_draft_tokens (which is auto-adjusted) or any other speculation parameters. It assumes the default auto-adjustment is optimal.

Broader Significance

This message is a microcosm of the entire optimization process documented in the session. It shows that the most impactful optimizations are not always the most complex ones. The assistant spent hours trying to autotune MoE kernels, patching scripts, and debugging build issues — only to find that the default kernel configuration was already near-optimal for this model. The real win came from a simple parameter change that took minutes to implement and test.

This is a lesson that applies broadly to ML inference optimization: understand your bottleneck before optimizing. The assistant correctly identified that the MoE kernel was not the bottleneck for this model (tiny expert matrices, memory-bound), and pivoted to speculation tuning, which directly addressed the actual throughput limiter.

The message also demonstrates the value of systematic benchmarking. Without the baseline numbers from [msg 6491], the assistant would have no way to evaluate the impact of the change. The careful methodology — same script, same concurrency levels, same measurement approach — ensures that the comparison is valid.

Conclusion

Message [msg 6502] is a masterclass in targeted performance optimization. It documents a hypothesis-driven change to the speculative decoding depth, measures both the benefits (up to 57% throughput improvement) and the costs (19% reduction in concurrent request capacity), and presents the results clearly. The assistant's ability to pivot from a dead-end optimization path (MoE kernel tuning) to a high-impact one (speculation depth) reflects deep system-level understanding and practical engineering judgment. For anyone deploying large language models with speculative decoding, this message offers a valuable lesson: sometimes the simplest parameter change yields the biggest gains, but only if you understand where the real bottleneck lies.