The Speculative Decoding Tuning Frontier: Pushing Qwen3.5-122B to 277 tok/s

In the high-stakes world of large language model inference, every token per second counts. When you're serving a 122-billion-parameter model with only 10 billion active parameters per forward pass (the A10B architecture), the difference between a mediocre deployment and a world-class one often comes down to how effectively you can exploit the model's architectural affordances. In message [msg 6515] of this opencode session, the assistant reaches a critical inflection point in its systematic exploration of speculative decoding parameters for the Qwen3.5-122B-A10B model running on four NVIDIA RTX PRO 6000 Blackwell GPUs. Having just benchmarked the configuration with speculative-num-steps=4, the assistant achieves an eye-popping 277 tok/s single-request throughput — a 125% improvement over the baseline — and must now decide whether to push further or declare the optimum found.

The Message: A Moment of Evaluation

The message itself is deceptively simple. It opens with an exclamation — "277 tok/s single-request!" — before immediately pivoting to a more cautious analytical posture: "But let me check — is the aggregate throughput at C=16+ starting to plateau or decrease?" This dual stance — celebration tempered by rigorous evaluation — characterizes the entire message. The assistant is not content to simply report a number; it must understand the shape of the performance landscape.

What follows is a four-column comparison table spanning five concurrency levels (C=1, 4, 16, 32, 64) across two metrics: per-request throughput and aggregate throughput. The table compares steps=1 (the baseline), steps=2, steps=3, and the newly measured steps=4. This table is the core analytical artifact of the message, and the assistant's interpretation of it reveals a sophisticated understanding of the tradeoffs inherent in speculative decoding.

The Pattern: Diminishing Returns and a Crossover

The data tells a clear story. At low concurrency (C=1-4), steps=4 is unambiguously superior. Single-request throughput climbs monotonically: 123 → 186 → 234 → 277 tok/s as steps increase from 1 to 4. That's a 2.25× improvement over the baseline — a remarkable gain from a single parameter change. At C=4, the pattern holds: 119 → 162 → 200 → 236 tok/s.

But at C=16, something interesting happens. The aggregate throughput for steps=4 drops to 1596 tok/s, below the steps=3 value of 1733 tok/s. At C=32, steps=4 achieves 1773 tok/s versus steps=3's 1979 tok/s. At C=64, the gap narrows but steps=4 still trails: 1817 vs 1914 tok/s.

The assistant immediately identifies the cause: "More draft tokens per step means each step takes longer, reducing batch efficiency at high concurrency. The max_running_requests is also lower (fewer concurrent requests can fit)." This is a crucial insight. Speculative decoding with more steps consumes more KV cache memory per request (because the draft tokens need to be cached), which reduces the total number of concurrent requests the system can handle. At high concurrency, the batch size is constrained by KV cache capacity, and the longer per-step computation time compounds the problem.

This creates a crossover point: steps=4 is optimal for latency-sensitive low-concurrency workloads, while steps=3 is better for throughput-oriented high-concurrency scenarios. The assistant's recommendation — "For agentic coding (typically 1-4 concurrent requests), steps=4 is clearly optimal" — is a pragmatic, use-case-driven conclusion.

The Decision: Push to Steps=5

Having identified the pattern, the assistant makes a deliberate decision: "But let me try steps=5 to see if we've hit the ceiling." This is not idle curiosity. The assistant has observed that each step increase has yielded substantial improvements at low concurrency, and it wants to know where the diminishing returns become severe enough to make further increases counterproductive even for single-request scenarios.

The decision is informed by the observation that steps=4 already shows degradation at high concurrency. If steps=5 continues to improve single-request throughput, it would be valuable for the primary use case. But if it plateaus or decreases, the assistant has found the ceiling. The command issued — stopping the service, killing any lingering processes, and freeing GPU memory — is the standard prelude to reconfiguring the systemd service file with the new parameter value.

Assumptions and Their Implications

The assistant's analysis rests on several assumptions that deserve scrutiny. The most significant is the benchmark methodology itself. The bench_qwen.py script uses temperature=0 and ignore_eos=True, which means the model generates greedily (deterministically) and never stops early. Under these conditions, the speculative decoding acceptance rate approaches 100% — the draft tokens are almost always accepted because there is no randomness in the generation. This is the best-case scenario for speculative decoding.

In real-world usage, particularly for creative or reasoning tasks, the temperature is typically >0, and the acceptance rate can be significantly lower. A lower acceptance rate means the extra speculation steps are more likely to be wasted computation. The optimal number of steps for a production deployment might be lower than what these benchmarks suggest. The assistant acknowledges this obliquely — "But this is with temperature=0 (greedy), which means ~100% acceptance rate" — in an earlier message ([msg 6509]) but does not adjust the methodology to account for it.

Another assumption is that the benchmark workload — fixed-length generation with a warmup phase — is representative of agentic coding patterns. Agentic coding involves a mix of short and long generations, tool calls, and structured outputs. The benchmark's steady-state throughput measurement may not capture the latency spikes or cold-start effects that matter in interactive use.

Input Knowledge: What You Need to Understand This Message

To fully grasp the significance of this message, one needs several pieces of contextual knowledge. First, the architecture of speculative decoding in SGLang: the speculative-num-steps parameter controls how many draft tokens the EAGLE (EAGLE-3) draft model generates per step. With topk=1, the number of draft tokens is steps + 1 (the original token plus speculated tokens). Each additional step increases the potential speedup but also increases the KV cache memory footprint and per-step computation time.

Second, the hardware context: four RTX PRO 6000 Blackwell GPUs with 96GB of HBM3e memory each, connected via NVLink. The total available KV cache memory determines max_running_requests, which drops from 26 (steps=1) to 21 (steps=2) to 17 (steps=3) as reported in earlier messages. This directly impacts high-concurrency throughput.

Third, the model architecture: Qwen3.5-122B-A10B uses a Mixture-of-Experts design where only 10 billion of the 122 billion parameters are active per token. This makes it particularly well-suited to speculative decoding because the draft model can be much smaller than the main model while still achieving high acceptance rates.

Output Knowledge: What This Message Creates

This message generates several important pieces of knowledge. The most concrete is the empirical performance data: a complete mapping of speculative-num-steps values (1-4) across five concurrency levels, with both per-request and aggregate throughput metrics. This data is immediately actionable for anyone deploying this model on similar hardware.

More abstractly, the message creates a decision framework for tuning speculative decoding parameters. The assistant demonstrates how to identify the crossover point where increased speculation hurts batch throughput, and how to match the parameter choice to the expected workload. The insight that "more draft tokens = longer step time = reduced batch efficiency" is a general principle that applies beyond this specific model and hardware combination.

The message also implicitly documents the relationship between speculative steps and KV cache memory consumption. The declining max_running_requests values (26 → 21 → 17) as steps increase is a critical system constraint that any production deployment must account for.

The Thinking Process: Methodical Exploration

What makes this message remarkable is not just the impressive throughput numbers, but the systematic thinking process behind them. The assistant has been running a controlled experiment across four parameter values, holding all other variables constant (model, hardware, batch size, benchmark script, temperature). Each step in the sequence — from steps=1 through steps=4 — was tested with the same methodology, producing comparable results.

The assistant's reasoning shows several hallmarks of good experimental practice. It doesn't just report the best number; it examines the full pattern across concurrency levels. It identifies the crossover and explains why it occurs. It connects the benchmark results to the practical use case. And it explicitly acknowledges the limitation of the greedy decoding assumption while still proceeding with the exploration.

The decision to try steps=5 is particularly telling. The assistant could have stopped at steps=4, declared victory with 277 tok/s, and moved on. Instead, it wants to find the true optimum — to understand where the ceiling is. This intellectual curiosity, combined with rigorous methodology, is what separates a superficial tuning exercise from a deep understanding of the system's behavior.

Conclusion

Message [msg 6515] captures a pivotal moment in a larger optimization journey. The assistant has pushed the Qwen3.5-122B-A10B model from 123 tok/s to 277 tok/s single-request throughput — a 2.25× improvement — through careful tuning of a single parameter. But more importantly, it has developed a nuanced understanding of the tradeoffs involved, identifying the crossover between low-concurrency and high-concurrency optimal configurations. The decision to push to steps=5 represents a commitment to finding the true ceiling, not just a good-enough result. Whether steps=5 yields further gains or reveals the plateau, the methodology and reasoning on display here provide a template for systematic performance optimization in complex ML inference systems.