The Diminishing Returns of Speculation: Benchmarking Steps=5 on Qwen3.5-122B

Message Overview

In this brief but consequential message ([msg 6519]), the assistant executes a single benchmark command against a production SGLang deployment of the Qwen3.5-122B-A10B BF16 model, running on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with speculative decoding (EAGLE-3) configured at speculative_num_steps=5. The output is a clean benchmark table showing aggregate and per-request throughput at four concurrency levels (1, 4, 16, 32 concurrent requests). The message itself contains no analysis — it is a raw data point, the latest in a systematic sweep across speculation depths that began with steps=1 and has now reached steps=5. The analysis will follow in the next message, but this message is the experimental result that reveals a critical inflection point in the speculation tuning curve.

[assistant] [bash] ssh root@10.1.230.174 '/root/ml-env/bin/python3 /root/bench_qwen.py 1 4 16 32' 2>&1
Warming up...
  Warmup: 100 tokens, ok=True

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      274.4 |          281.7 |     4000 |     14.6 |    4
   4 |      785.1 |          221.6 |     7396 |      9.4 |    8
  16 |     1119.4 |          186.8 |    29672 |     26.5 |   32
  32 |     1273.3 |          190.7 |    62000 |     48.7 |   64
-----------------------------------------------------------------

Context: The Speculation Tuning Campaign

To understand why this message matters, one must trace the arc of the preceding 20+ messages. The assistant had been engaged in a multi-day effort to optimize inference throughput for the Qwen3.5-122B-A10B model — a 122-billion-parameter MoE (Mixture-of-Experts) model with 10 billion active parameters per token. The deployment target was a Proxmox-hosted VM with RTX PRO 6000 Blackwell GPUs, running SGLang with EAGLE-3 speculative decoding.

Earlier in the session ([msg 6482]–[msg 6494]), the assistant had explored MoE Triton kernel autotuning, copying B200 kernel configurations and testing various SGLang flags like --enable-fused-moe-sum-all-reduce and --enable-flashinfer-allreduce-fusion. None of these produced measurable improvements — the MoE kernel was not the bottleneck for this model, because with E=256 experts and N=256 intermediate size, the expert GEMM operations are memory-bound on tiny matrices.

The breakthrough came when the assistant shifted focus to speculative decoding parameters. Starting from a baseline of speculative_num_steps=1 (which produced ~123 tok/s single-request throughput), the assistant systematically increased the speculation depth:

Steps=1: 123 tok/s per-req (baseline)
Steps=2: 186 tok/s per-req (+51%)
Steps=3: 234 tok/s per-req (+90%)
Steps=4: 277 tok/s per-req (+125%) Each increase yielded substantial gains at low concurrency (C=1), though the aggregate throughput at high concurrency (C=16+) began to plateau starting at steps=4. The tradeoff is structural: more speculation steps mean more draft tokens per verification step, which increases the KV cache footprint per request and reduces max_running_requests (the number of concurrent requests the server can accommodate). At steps=1, the server could run 26 concurrent requests; at steps=4, that dropped to approximately 14. Message [msg 6519] represents the steps=5 data point — the logical next step in this sweep, testing whether the trend of improving single-request throughput continues or whether diminishing returns have set in.

What the Data Reveals

The benchmark results tell a nuanced story. At C=1 (single request), steps=5 achieves 274.4 aggregate tok/s and 281.7 per-request tok/s. Compared to steps=4's 270.4 agg / 277.4 per-req, this is a marginal improvement of roughly 1.5% — barely outside noise. The single-request throughput has effectively plateaued.

At C=4, steps=5 actually underperforms steps=4: 785.1 agg vs 830.3 agg, a 5.4% regression. At C=16, the regression is dramatic: 1119.4 agg vs 1595.7 agg, a 30% drop. At C=32: 1273.3 agg vs 1773.0 agg, a 28% drop.

The pattern is clear: steps=5 offers no meaningful benefit at any concurrency level and significantly degrades throughput at higher concurrency. The speculation depth has crossed the optimal point.

The Reasoning Behind the Benchmark

This message was written to answer a specific question: Does increasing speculative steps from 4 to 5 continue to improve throughput, or have we hit the ceiling? The assistant had already observed at steps=4 that aggregate throughput at C=16+ was starting to plateau compared to steps=3 (1596 vs 1733 agg at C=16). The hypothesis was that steps=5 might push single-request throughput even higher — perhaps to 300+ tok/s — while further degrading high-concurrency throughput.

The assistant's thinking process, visible in the preceding messages, shows a careful cost-benefit analysis. Each increase in speculative_num_steps carries two costs: (1) longer per-step computation time (more draft tokens to verify), and (2) reduced max_running_requests due to KV cache pressure. The benefit is higher speculative acceptance — with more draft tokens, the probability of accepting a correct continuation increases, especially for greedy (temperature=0) decoding where the model's output is deterministic and highly predictable.

The assistant had also noted a critical detail in [msg 6501]: SGLang automatically adjusts speculative_num_draft_tokens to speculative_num_steps + 1 when speculative_eagle_topk == 1. So steps=5 means 6 draft tokens per verification step. Each verification step must process all 6 draft tokens, which increases the per-step latency. If the acceptance rate is already near 100% (as it would be for greedy decoding of repetitive benchmark prompts), additional draft tokens provide no benefit — the model would have accepted the first 5 tokens anyway.

Assumptions and Limitations

Several assumptions underpin this benchmark. First, the benchmark uses temperature=0, ignore_eos=True, which produces highly repetitive, predictable output. This artificially inflates the speculative acceptance rate — the model is generating the same patterns repeatedly, making it trivially easy for the draft model to predict. In real agentic workloads with diverse, creative text generation, acceptance rates would be lower, and the optimal speculation depth might differ.

Second, the benchmark uses a fixed prompt and generation length. The bench_qwen.py script (seen in earlier messages) sends requests that generate a fixed number of tokens. Real workloads have variable-length inputs and outputs, which would change the KV cache dynamics.

Third, the benchmark skips C=64 (compared to earlier runs which included it). This may have been intentional — the assistant may have judged that the trend was already clear from C=1–32, or that the reduced max_running_requests at steps=5 made C=64 impossible to measure reliably.

Input Knowledge Required

To interpret this message, the reader needs to understand:

Speculative decoding (EAGLE-3): A technique where a lightweight draft model proposes multiple candidate tokens, and the main model verifies them in parallel. The num_steps parameter controls how many draft tokens are generated per verification cycle.
The benchmark format: The table shows concurrency level (C), aggregate throughput (Agg tok/s — total tokens generated across all requests per second), per-request throughput (Per-req tok/s — average tokens per second for each individual request), total tokens generated, wall clock time, and number of requests that completed successfully (Ok).
The hardware context: RTX PRO 6000 Blackwell GPUs with 96GB VRAM each, running SGLang with EAGLE-3 speculation. The model is Qwen3.5-122B-A10B, a 122B-parameter MoE model with 10B active parameters.
The KV cache tradeoff: More speculation steps means more draft tokens in flight, which consumes KV cache memory and reduces the number of concurrent requests the server can handle.

Output Knowledge Created

This message creates a critical data point in the speculation tuning curve. It demonstrates that:

Single-request throughput peaks at approximately steps=4–5, with steps=5 offering no meaningful improvement over steps=4 (~277 vs ~282 tok/s).
High-concurrency throughput (C=16+) degrades significantly at steps=5, with aggregate throughput dropping 28–30% compared to steps=4.
The optimal speculation depth for this model/hardware combination is steps=3–4, depending on whether single-request latency or aggregate throughput is the priority. For agentic coding workloads (typically 1–4 concurrent requests), steps=4 appears optimal, offering 277 tok/s single-request with reasonable aggregate throughput. For high-throughput serving (16+ concurrent requests), steps=3 is better, offering 1733 agg tok/s at C=16 versus steps=4's 1596.

The Broader Significance

This message exemplifies a pattern that recurs throughout the session: systematic, data-driven optimization. The assistant does not guess at optimal parameters — it runs controlled experiments, varies one parameter at a time, and compares results quantitatively. The speculation sweep from steps=1 through steps=5, with each step benchmarked at multiple concurrency levels, is a textbook example of performance tuning methodology.

The message also illustrates a fundamental principle of speculative decoding: the optimal speculation depth is not a universal constant but a function of the workload, the model, the hardware, and the serving configuration. Too little speculation leaves throughput on the table; too much speculation wastes compute on draft tokens that are never accepted and consumes memory that could serve other requests. Finding the sweet spot requires exactly this kind of empirical sweep.

In the next message ([msg 6520]), the assistant will immediately analyze these results and conclude: "Steps=5 gives 282 tok/s at C=1 — only marginally better than steps=4 (277). And aggregate throughput drops significantly at C=16+ (1119 vs 1596). Diminishing returns." The user then proposes an even more aggressive test ("Try 10 steps to see if we unplateu"), but the data already tells a clear story: the speculation curve has flattened, and further increases will only degrade performance.