The Pivot That Paid Off: Tuning Speculative Decoding for Qwen3.5-122B on Blackwell GPUs

A Single Message That Changed the Trajectory

In the course of a long and technically demanding optimization session, there comes a moment when the engineer steps back, re-evaluates the evidence, and makes a strategic pivot. Message [msg 6495] captures exactly such a moment. It is deceptively short — just two sentences of reasoning followed by a single bash command — but it represents the culmination of an extended investigation into MoE kernel autotuning and a decisive shift toward a far more fruitful optimization direction.

The message reads:

Let me try increasing --speculative-num-steps from 1 to 2. With num-steps=1 and topk=1, we draft 2 tokens per step (1 original + 1 speculated). With num-steps=2 and topk=1, we'd draft 3 tokens per step. This should improve throughput if the acceptance rate is decent. [bash] ssh root@10.1.230.174 'systemctl stop sglang-qwen.service' 2>&1

To understand why this message matters, we must examine the context that led to it, the reasoning it encodes, the assumptions it makes, and the dramatic results it ultimately produced.

The Context: A Dead End in MoE Kernel Tuning

The assistant had spent the preceding messages (from [msg 6472] onward) pursuing an entirely different optimization strategy: autotuning the fused MoE Triton kernels for the RTX PRO 6000 Blackwell GPUs (SM120 architecture). The model in question, Qwen3.5-122B-A10B, is a Mixture-of-Experts architecture with 256 experts, a per-expert intermediate size of just 256, and a hidden size of 3072. The assistant had copied configuration files from the NVIDIA B200 (a similar Blackwell GPU) and run benchmarks, only to find "Same numbers within noise" ([msg 6491]).

This prompted a crucial moment of analysis in [msg 6494]. The assistant examined the default kernel configuration code and realized that for BF16 with small M values (where M ≤ E=256), the default parameters were already quite close to the B200-optimized config. More importantly, the assistant identified the root cause: "With E=256 experts but only N=256 per-expert intermediate size (very small matrices per expert), the GEMM is memory-bound not compute-bound. The kernel config doesn't matter much when the matrices are tiny."

This insight — that the MoE GEMM kernels were not the bottleneck — is the intellectual foundation for message [msg 6495]. The assistant correctly diagnosed that the tiny matrix dimensions (256×3072 per expert) meant the computation was fundamentally memory-bound, and no amount of kernel tuning would meaningfully change throughput. This is a textbook example of understanding the hardware-computation interaction: when matrices are small enough to fit in on-chip SRAM, the kernel launch overhead and memory bandwidth dominate, not the arithmetic intensity.

The Reasoning: Why Speculative Num-Steps Matters

Having correctly identified that MoE kernel tuning was a dead end, the assistant pivoted to "the speculative decoding tuning which has a more direct impact on per-request latency." Message [msg 6495] is the first concrete action of this new strategy.

The reasoning is precise and shows deep understanding of how SGLang's EAGLE-3 speculative decoding works. The assistant is running with --speculative-eagle-topk 1, which means the draft model proposes exactly one candidate token per speculation step. The key parameter being changed is --speculative-num-steps, which controls how many speculative steps are taken per decoding iteration.

The assistant's mental model is:

• With num_steps=1 and topk=1: Each iteration produces 1 verified token (the original) + 1 speculated token that may be accepted = up to 2 tokens per step.
• With num_steps=2 and topk=1: Each iteration produces 1 verified token + 2 speculated tokens = up to 3 tokens per step. The assumption is that the acceptance rate — the probability that the draft model's proposed tokens match what the target model would have generated — is "decent." If acceptance is high, increasing the number of speculation steps directly increases throughput because more tokens are generated per model invocation. If acceptance is low, the extra speculation steps would be wasted computation, and the overhead of the draft model's forward passes could even hurt performance. This is a calculated risk. The assistant doesn't know the acceptance rate a priori — it depends on the specific model, the task distribution, and the quality of the EAGLE-3 draft module. The assumption that the acceptance rate is "decent" is grounded in the fact that EAGLE-3 is specifically designed for this model family (it was trained on Qwen3.5's own hidden states), but it remains an empirical question.

The Input Knowledge Required

To understand and execute this optimization, the assistant needed a substantial body of knowledge:

1. SGLang's speculative decoding architecture: Understanding that --speculative-num-steps controls the number of draft tokens generated per iteration, and that with topk=1 the draft tokens are adjusted to num_steps + 1 (as confirmed in [msg 6501]).
2. The relationship between speculation parameters and KV cache budget: More draft tokens consume more KV cache slots per request, reducing the maximum number of concurrent requests. The assistant would later discover that max_running_requests dropped from 26 to 21 and max_total_num_tokens dropped from 455,357 to 377,207 ([msg 6502]).
3. The model's architecture: Qwen3.5-122B-A10B with its 256 experts, tiny per-expert matrices, and the fact that the GEMM kernel was not the bottleneck.
4. The deployment infrastructure: The systemd service file location, the service management commands, and the SSH access to the remote machine (10.1.230.174).
5. The benchmarking methodology: The assistant had a bench_qwen.py script ready and knew how to interpret its output.

The Output Knowledge Created

Message [msg 6495] itself only produces one concrete output: the service is stopped. But it sets in motion a chain of events that generates substantial new knowledge:

• The systemd service file is edited (<msg id=6497-6498>) to add --speculative-num-steps 2 to the SGLang server arguments.
• The server is restarted and its logs confirm the new parameters: speculative_num_steps=2, speculative_num_draft_tokens=3 ([msg 6501]).
• Benchmark results ([msg 6502]) show dramatic improvement: single-request throughput jumps from 122.9 tok/s to 182.7 tok/s — a 49% improvement. At concurrency 64, aggregate throughput increases from 1566.1 tok/s to 1844.0 tok/s — an 18% improvement. These results validate the assistant's hypothesis. The acceptance rate was indeed decent enough that the extra speculation step produced a net gain. The trade-off — reduced maximum concurrency due to higher KV cache consumption — was worth it for the throughput improvement.

The Thinking Process Visible in the Reasoning

What makes this message particularly interesting is what it reveals about the assistant's decision-making process. The assistant had just spent considerable effort on MoE kernel tuning, including patching the autotuning script ([msg 6478]), running a long tuning job that timed out ([msg 6479]), copying B200 configs ([msg 6483]), and benchmarking ([msg 6491]). The ability to recognize when a line of investigation is not paying off and to pivot decisively is a hallmark of effective engineering.

The assistant's reasoning in [msg 6494] shows a clear analytical chain:

1. The B200 config didn't improve performance.
2. The default config is already close to optimal for small M.
3. The fundamental reason: tiny matrices make the kernel memory-bound.
4. Therefore, focus on speculative decoding instead. Message [msg 6495] then executes this pivot with a specific, testable hypothesis. The assistant doesn't just vaguely suggest "tuning speculative decoding" — it identifies a concrete parameter change, explains the expected mechanism (more draft tokens per step), and states the condition for success (decent acceptance rate). This is hypothesis-driven optimization at its best.

The Broader Significance

This message exemplifies a pattern that recurs throughout the session: the assistant systematically eliminates non-bottlenecks and converges on the actual performance lever. Earlier, it had tried --enable-fused-moe-sum-all-reduce and --enable-flashinfer-allreduce-fusion (both documented in the todo list at the end of [msg 6494]), found them ineffective, and moved on. Message [msg 6495] represents the culmination of this elimination process and the beginning of the most impactful optimization of the entire deployment.

The 49% single-request throughput improvement that resulted from this single parameter change — from 122.9 to 182.7 tok/s — is a dramatic demonstration that understanding the bottleneck is far more valuable than blindly tuning parameters. The assistant's willingness to abandon a line of investigation that had consumed significant effort (patching scripts, running autotuning, copying configs) and pivot to a different approach is a lesson in intellectual honesty and engineering pragmatism.

In the end, message [msg 6495] is a small message with large consequences — a strategic pivot disguised as a routine parameter tweak, and a testament to the value of understanding what actually limits performance before attempting to optimize.