The Power of Speculation: Pushing Multi-Token Prediction to Its Limits on Blackwell

Introduction

In the world of large language model inference, every token per second counts. For a 122-billion-parameter model like Qwen3.5-122B-A10B running on four NVIDIA RTX PRO 6000 Blackwell GPUs, achieving high throughput requires careful orchestration of model parallelism, kernel selection, and — perhaps most importantly — speculative decoding. Message <msg id=6514> captures the triumphant conclusion of a systematic optimization campaign: a benchmark run showing 277.4 tokens per second for a single request, more than double the baseline performance from just a few iterations earlier.

This message, at first glance, is simply the output of a benchmark script. But it represents the culmination of a deliberate, hypothesis-driven tuning process where the assistant progressively increased the speculative-num-steps parameter — the number of tokens the draft model predicts ahead per step — from 1 to 4, measuring the impact at each step. The results are striking and tell a compelling story about the power of aggressive speculation when conditions are favorable.

The Message: Raw Benchmark Output

The message consists of a single bash command and its output:

ssh root@10.1.230.174 '/root/ml-env/bin/python3 /root/bench_qwen.py 1 4 16 32 64' 2>&1

The benchmark script tests five concurrency levels (1, 4, 16, 32, and 64 simultaneous requests) and reports aggregate throughput, per-request throughput, total tokens generated, wall-clock time, and request success count. The results for speculative-num-steps=4 are:

| Concurrency | Aggregate tok/s | Per-request tok/s | Tokens | Wall time (s) | |-------------|----------------|-------------------|--------|---------------| | 1 | 270.4 | 277.4 | 4,000 | 14.8 | | 4 | 830.3 | 235.5 | 7,441 | 9.0 | | 16 | 1,595.7 | 163.4 | 29,798 | 18.7 | | 32 | 1,773.0 | 154.9 | 59,000 | 33.3 | | 64 | 1,817.1 | 163.2 | 112,000| 61.6 |

The Ok column shows all requests completed successfully (4, 8, 32, 64, 128 respectively — each concurrency level runs multiple rounds), confirming the server remained stable under load.

The Optimization Journey: Why This Message Was Written

This message was not written in isolation. It is the fourth in a sequence of benchmarks where the assistant systematically increased the speculative-num-steps parameter and measured the effect. Understanding why this message exists requires tracing the reasoning that led to it.

The Problem: Stagnant MoE Kernel Performance

Earlier in the session, the assistant had attempted to improve throughput by tuning the fused MoE (Mixture-of-Experts) Triton kernel. The model uses 256 experts with a per-expert intermediate size of just 256 neurons — tiny matrices that make the GEMM operations memory-bound rather than compute-bound. The assistant copied B200-optimized kernel configurations, ran autotuning, and tried various fusion flags (--enable-fused-moe-sum-all-reduce, --enable-flashinfer-allreduce-fusion). None produced measurable improvements.

After hitting diminishing returns on kernel optimization, the assistant made a critical pivot. In message <msg id=6494>, the reasoning is explicit:

"Let me step back and think about what would actually make a difference. The MoE kernel for this model has E=256, N=256... Each expert GEMM is 256x3072 (tiny). The real bottleneck is likely the expert selection and token routing, not the GEMM kernel itself. Let me instead focus on the speculative decoding tuning which has a more direct impact on per-request latency."

This insight — recognizing that the bottleneck had shifted from compute to memory-bound operations and that speculative decoding offered more leverage — drove the entire subsequent optimization campaign.

The Hypothesis: More Speculation Steps → Higher Throughput

The assistant's hypothesis was straightforward: with speculative_eagle_topk=1 (drafting a single token sequence rather than a tree), the number of draft tokens per step equals speculative_num_steps + 1. Increasing speculative_num_steps from 1 to 4 would increase the draft length from 2 to 5 tokens per speculative step, potentially increasing throughput if the acceptance rate remained high.

The assistant tested this hypothesis methodically:

Assumptions and Their Implications

The assistant made several assumptions during this optimization, some explicit and some implicit:

1. Greedy Decoding (Temperature=0) Maximizes Acceptance

The benchmark script uses temperature=0, ignore_eos=True, which makes the model deterministic and highly predictable. Under these conditions, the draft model's predictions match the target model's output with near-100% acceptance rate. This is the friendliest possible scenario for speculative decoding.

Implication: The 277 tok/s figure represents an upper bound, not a realistic deployment number. In production, with non-zero temperature and varied prompts, acceptance rates would be lower, and the optimal number of speculation steps might differ. The assistant acknowledged this concern in <msg id=6503>:

"But wait, this benchmark uses temperature=0, ignore_eos=True which artificially inflates acceptance rates (repetitive output is highly predictable). Let me also check with a more realistic workload."

2. The KV Cache Budget Is Sufficient

Increasing speculative_num_steps from 1 to 4 reduced max_running_requests from 26 to approximately 17 (observed at steps=3) because each request consumes more KV cache slots for draft tokens. The assistant monitored this trade-off but did not explicitly test whether the reduced batch size at high concurrency was limiting throughput.

3. The Draft Model Quality Is Consistent Across Steps

The assistant implicitly assumed that the draft model's prediction quality (trained as part of the Qwen3.5 model's MTP head) remains high for longer speculation horizons. This is a reasonable assumption for the specific model, which was trained with multi-token prediction, but it may not generalize to other models or fine-tuned variants.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this benchmark output, one needs:

  1. Understanding of speculative decoding (MTP): The mechanism where a lightweight draft model predicts multiple future tokens, and the main model verifies them in parallel. The speculative-num-steps parameter controls how many tokens ahead the draft predicts.
  2. Knowledge of the hardware: Four NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture) with InfiniBand interconnect, running Qwen3.5-122B-A10B — a 122B-parameter MoE model with 256 experts and 10B active parameters.
  3. Context of the optimization journey: The assistant had already ruled out MoE kernel tuning and allreduce fusion as viable optimization paths, making speculative decoding the primary lever.
  4. Familiarity with the benchmark methodology: The bench_qwen.py script uses fixed prompt templates, temperature=0, and ignore_eos=True, which creates best-case conditions for speculative decoding.

Output Knowledge Created by This Message

This benchmark run produced several valuable insights:

  1. Empirical upper bound for single-request throughput: 277 tok/s on 4× Blackwell GPUs for a 122B MoE model. This is a reference point for anyone deploying similar hardware.
  2. Confirmation of the speculation scaling trend: The improvement from steps=1 to steps=4 was not linear but showed continued gains at each increment (50%, 88%, 125% relative to baseline). This suggests the model's MTP head is well-trained for multi-step prediction.
  3. Diminishing returns at high concurrency: At C=64, the improvement from steps=3 to steps=4 was modest (1,914 → 1,817 agg tok/s — actually a slight regression). The aggregate throughput peaked at steps=3 for high concurrency, suggesting that the KV cache pressure from longer draft sequences begins to hurt batch throughput.
  4. The per-request throughput sweet spot: At C=1, per-request throughput increased monotonically with steps. At higher concurrency, the per-request throughput plateaued around 155-163 tok/s regardless of steps, indicating that the server becomes compute-bound rather than speculation-bound under load.

The Thinking Process: A Window Into Systematic Optimization

The assistant's reasoning, visible across the sequence of messages, reveals a disciplined optimization methodology:

Step 1: Identify the bottleneck. Rather than blindly tuning parameters, the assistant analyzed why MoE kernel optimization was ineffective — the matrices are too small for kernel configuration to matter. This required understanding both the model architecture (E=256, N=256) and the hardware characteristics (memory-bound vs compute-bound regimes).

Step 2: Form a hypothesis. The assistant hypothesized that speculative decoding, which directly reduces the number of serial autoregressive steps, would have a larger impact than kernel micro-optimizations.

Step 3: Test incrementally. Rather than jumping to an extreme setting, the assistant tested steps=2, then steps=3, then steps=4, benchmarking each configuration with the same workload. This allowed measuring marginal returns at each step.

Step 4: Monitor side effects. The assistant checked system logs for KV cache adjustments (max_total_num_tokens, max_running_requests) to understand the resource trade-offs of longer speculation.

Step 5: Acknowledge limitations. The assistant explicitly noted that the benchmark conditions (temperature=0) are unrealistic and that real-world performance would likely be lower.

This thinking process is a model of how to approach performance optimization in complex systems: understand the bottleneck, form a hypothesis, test incrementally, monitor side effects, and acknowledge the gap between benchmark and production conditions.

Mistakes and Incorrect Assumptions

While the optimization was successful, there are some potential issues worth noting:

  1. No realistic workload test was conducted. The assistant acknowledged the temperature=0 issue but did not follow up with a more realistic benchmark within this sequence. The 277 tok/s figure could mislead someone into expecting similar performance in production.
  2. The regression at C=64 for steps=4 vs steps=3 was not investigated. The aggregate throughput dropped from 1,914 tok/s (steps=3) to 1,817 tok/s (steps=4) at the highest concurrency. This could indicate that the KV cache pressure from longer draft sequences is hurting batch efficiency, but the assistant did not explore this.
  3. No ablation of other parameters. The assistant only varied speculative-num-steps. Other parameters like speculative-eagle-topk (tree-based speculation), draft model batch size, or the interaction with tensor parallelism were not explored.

Conclusion

Message <msg id=6514> is more than a benchmark result — it is the culmination of a systematic optimization campaign that demonstrates the transformative power of speculative decoding when applied to a model with a well-trained MTP head. The 277 tok/s single-request throughput represents a 125% improvement over the baseline, achieved through disciplined hypothesis testing and incremental parameter tuning.

The message also serves as a case study in knowing where to focus optimization effort. The assistant correctly identified that MoE kernel tuning was a dead end for this particular model and pivoted to speculative decoding — the lever that actually moved the needle. This ability to recognize when a line of inquiry has diminishing returns and to redirect effort is perhaps the most valuable skill demonstrated in this exchange.

For anyone deploying large MoE models on Blackwell hardware, these results provide a valuable reference point: with aggressive speculation (steps=4) and favorable conditions (greedy decoding), a 122B model can achieve throughput that rivals much smaller models, making it viable for latency-sensitive applications like agentic coding where single-request response time matters most.