The Power of Speculative Depth: Benchmarking MTP Steps on Qwen3.5-122B

Introduction

In the high-stakes world of large language model inference, every token per second counts. When serving a 122-billion-parameter Mixture-of-Experts model like Qwen3.5-122B-A10B across four NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between a usable interactive experience and a frustratingly slow one can come down to a single configuration parameter. Message 6509 in this opencode session captures a pivotal moment: the assistant has just received benchmark results from testing three different speculative decoding configurations and is faced with a decision about how far to push the parameter.

This message is a study in data-driven optimization. It contains a carefully constructed comparison table, a clear analytical conclusion, and a decisive next step. More than just a status update, it represents the culmination of a multi-hour investigation into speculative decoding tuning, where the assistant has systematically explored how the number of MTP (Multi-Token Prediction) steps affects throughput across different concurrency levels. The results are striking—a 90% improvement in single-request throughput—and the message captures the excitement and analytical rigor of that discovery.

The Context: A Long Optimization Journey

To understand why this message matters, we need to appreciate the journey that led to it. The assistant had been working for hours on deploying and optimizing the Qwen3.5-122B-A10B model, a massive MoE architecture with 256 experts and a per-expert intermediate size of just 256 neurons. Earlier in the session, the assistant had explored MoE kernel autotuning, copying B200 configuration files, and testing various allreduce fusion flags—all with negligible impact on performance. The bottleneck, as the assistant correctly diagnosed, was not the GEMM kernels but the speculative decoding pipeline.

The breakthrough came when the assistant increased --speculative-num-steps from 1 to 2 (see [msg 6502]), yielding a 36-56% improvement in per-request throughput. Message 6509 follows immediately after testing steps=3, and it presents the comparison that confirms the trend is real and substantial.

What the Message Contains

The message opens with an enthusiastic "Even better!" and presents a detailed comparison table showing throughput across five concurrency levels (1, 4, 16, 32, and 64 concurrent requests) for three configurations: Steps=1, Steps=2, and Steps=3. The table shows both per-request throughput (the latency-sensitive metric for interactive use) and aggregate throughput (the total tokens per second across all requests).

The numbers tell a compelling story. At a single concurrent request—the most relevant scenario for interactive coding assistants and chat applications—throughput jumps from 123 tok/s (Steps=1) to 186 tok/s (Steps=2) to 234 tok/s (Steps=3), a 90% improvement over the baseline. Even at high concurrency (64 requests), per-request throughput nearly doubles from 66 to 120 tok/s.

The aggregate throughput tells a different but complementary story. At low concurrency, the improvement is dramatic (88% at C=1), but at high concurrency the gains compress to 21% (from 1582 to 1914 tok/s). This makes intuitive sense: speculative decoding helps most when there's idle GPU capacity to run the draft model, and at high concurrency the GPU is already saturated with batch processing.

The assistant then makes a critical observation: "But this is with temperature=0 (greedy), which means ~100% acceptance rate." This is an important caveat—the benchmark uses greedy decoding, which makes the speculative draft tokens almost always accepted because the output is deterministic and predictable. In real workloads with sampling, the acceptance rate would be lower, and the real-world gains would be smaller. The assistant acknowledges this limitation but continues the exploration, treating the current results as an upper bound on potential improvement.

The message concludes with a decision: "Let me push to steps=4 to see if we hit diminishing returns," followed by a bash command to stop the service for reconfiguration.

The Reasoning and Decision-Making Process

The assistant's thinking in this message is remarkably clear and structured. There are several layers of analysis visible:

Comparative analysis: The assistant doesn't just report the new numbers—they construct a three-way comparison table that contextualizes the results against previous benchmarks. This shows a methodical approach to experimentation where each new data point is evaluated relative to prior results.

Multi-metric evaluation: The table includes both per-request and aggregate throughput, recognizing that different stakeholders care about different metrics. For interactive use cases (the primary target for this deployment), per-request throughput at low concurrency is the key metric. For batch processing or high-throughput serving, aggregate throughput matters more.

Caveat identification: The assistant explicitly flags the temperature=0 limitation, showing awareness of the gap between benchmark conditions and real-world usage. This is not a naive "bigger numbers = better" conclusion but a nuanced understanding of what the numbers actually mean.

Diminishing returns consideration: The decision to try steps=4 is framed as a test for diminishing returns, not an assumption that more steps are always better. The assistant recognizes that each additional speculation step consumes KV cache memory (reducing max_running_requests from 26 to 21 to 17 across the three configurations) and that at some point the marginal gain will not justify the memory cost.

Assumptions and Their Implications

The message rests on several assumptions, some explicit and some implicit:

Greedy decoding as proxy: The most significant assumption is that temperature=0 benchmarks are representative of real performance. The assistant acknowledges this but proceeds anyway. This is a reasonable approach for initial tuning—establish the upper bound, then test with sampling later. However, it means the 90% improvement figure should be treated as aspirational rather than guaranteed.

Linear scaling of acceptance rate: The assistant implicitly assumes that the acceptance rate remains high across all three step configurations. In practice, longer speculation horizons tend to have lower acceptance rates for later tokens because the draft model's predictions become less accurate further into the future. The fact that steps=3 still shows strong gains suggests the acceptance rate remains high for this particular model and task, but this might not hold for all workloads.

Benchmark workload representativeness: The benchmark uses ignore_eos=True, meaning it generates a fixed number of tokens without stopping. This is common for throughput benchmarking but doesn't reflect real usage where the model generates variable-length responses and stops at natural boundaries.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. Speculative decoding: The technique of using a small draft model to predict multiple tokens, which are then verified by the main model in parallel. The "steps" parameter controls how many future tokens are speculated per iteration.
  2. MTP (Multi-Token Prediction): A specific form of speculative decoding where the draft model is integrated into the main model's architecture rather than being a separate model. In this deployment, it's the EAGLE-style MTP implementation in SGLang.
  3. The Qwen3.5-122B-A10B architecture: A Mixture-of-Experts model with 122B total parameters, 10B active parameters per token, 256 experts, and an intermediate size of 256 per expert. The small per-expert size means the MoE GEMM kernels are memory-bound rather than compute-bound.
  4. The hardware platform: Four NVIDIA RTX PRO 6000 Blackwell GPUs with NVLink interconnect, running on Ubuntu 24.04 with CUDA 13.0.
  5. The benchmark methodology: The bench_qwen.py script that warms up the model and then measures throughput at various concurrency levels with fixed-length generation.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. Quantified scaling laws for MTP steps: A clear empirical relationship showing that increasing MTP steps from 1 to 3 yields 41-90% improvement in per-request throughput, with diminishing returns beginning to appear at high concurrency.
  2. Configuration guidance: Evidence that --speculative-num-steps=3 is significantly better than the default of 1 for this model and hardware combination, providing a concrete recommendation for deployment.
  3. Memory-throughput tradeoff data: The earlier messages in this sequence showed that each additional step reduces the maximum number of concurrent requests (from 26 to 21 to 17). This message provides the throughput side of that tradeoff, enabling an informed decision about the optimal configuration.
  4. A benchmark methodology: The approach of testing multiple concurrency levels and reporting both per-request and aggregate throughput establishes a reusable evaluation framework.

The Thinking Process

What makes this message particularly interesting is the visible thinking process. The assistant doesn't just report numbers—they interpret them, caveat them, and act on them. The progression from "Steps=2 is good" to "Steps=3 is even better" to "Let me try Steps=4" shows a hypothesis-driven experimental approach.

The assistant is also clearly excited by the results. The "Even better!" opening and the bold formatting of the 90% improvement figure convey genuine enthusiasm. This is not a dispassionate log entry but the work of someone who has been wrestling with a performance problem and has finally found a lever that works.

At the same time, the assistant maintains scientific rigor. The caveat about temperature=0 shows they're not getting carried away by the numbers. The decision to test steps=4 is framed as a search for diminishing returns, not an assumption that more is always better. This balance of enthusiasm and rigor is characteristic of effective optimization work.

Conclusion

Message 6509 captures a breakthrough moment in a long optimization session. The assistant has discovered that increasing MTP speculation steps from 1 to 3 nearly doubles single-request throughput for the Qwen3.5-122B-A10B model on Blackwell GPUs, jumping from 123 to 234 tok/s. The message presents this finding with clear data, appropriate caveats, and a well-reasoned plan for further exploration.

What makes this message worth studying is not just the impressive numbers but the methodology behind them: the systematic comparison across multiple configurations and concurrency levels, the awareness of benchmark limitations, and the data-driven decision-making. It's a textbook example of how to approach inference optimization—measure carefully, compare systematically, caveat honestly, and iterate intelligently.