The MTP Benchmark: When Speculative Decoding Both Wins and Loses

In the course of deploying a production inference server for the Qwen3.5-122B-A10B model across four NVIDIA RTX PRO 6000 Blackwell GPUs, one message stands out as a moment of clear-eyed empirical reckoning. Message [msg 6366] is the assistant's analysis of a benchmark comparison between two configurations of the SGLang inference engine: one running with Multi-Token Prediction (MTP) speculative decoding enabled, and one without it. The message is deceptively simple — a markdown table and a few paragraphs of commentary — but it encapsulates a deep tension in modern LLM serving: speculative decoding techniques that improve per-request latency can simultaneously degrade aggregate throughput under high concurrency, forcing operators to make nuanced tradeoffs based on their actual workload.

The Context: Enabling MTP on a Hybrid Mamba-Attention Model

To understand the significance of this message, we must first understand what led to it. The assistant had spent the preceding session enabling MTP speculation — a form of speculative decoding where a lightweight "draft" model predicts multiple future tokens in a single forward pass, which the base model then verifies in parallel. For Qwen3.5-122B-A10B, a hybrid architecture combining Mamba state-space model layers with traditional attention layers, SGLang supports this through its --speculative-algorithm NEXTN flag, which the assistant had configured in the systemd service file ([msg 6352]–[msg 6355]).

The initial signs were promising. After restarting the server, the assistant observed in [msg 6363] that MTP was achieving an accept length of 2.00 tokens per step — meaning the draft model's predictions were being accepted at nearly 100% rate on simple queries — and single-request throughput had jumped from ~110 tok/s to 135 tok/s, a ~23% improvement. The draft model itself (Qwen3.5ForCausalLMMTP) consumed 1.91 GB of GPU memory per rank, which necessitated reducing mem_fraction_static from a higher value to 0.75.

But a single-request smoke test is not a production benchmark. The assistant needed to understand how MTP behaved under load — at varying levels of concurrent requests — and that is exactly what message [msg 6366] delivers.

The Benchmark: A Tale of Two Metrics

The assistant ran the existing benchmark script (bench_qwen.py) at concurrency levels of 1, 4, 16, 32, 64, and 128 concurrent requests, each generating 1000 output tokens from 1000 input tokens. The results, presented in a carefully constructed comparison table, tell a story that depends entirely on which metric you prioritize.

At low concurrency (C=1–4), MTP is a clear winner. Single-request throughput goes from 108 to 121.5 tok/s aggregate (+12%), and per-request throughput from 110 to 123.1 tok/s (+12%). At C=4, the improvement is even more dramatic: aggregate throughput jumps from 286 to 400.5 tok/s (+40%), and per-request from 82 to 118.6 tok/s (+45%). This is the sweet spot for MTP — when the server has idle compute capacity, the draft model's extra forward passes are essentially free, and the speculation accelerates every request.

At medium concurrency (C=16–32), the picture becomes more complex. Aggregate throughput at C=16 improves from 873 to 1,209 tok/s (+38%), and per-request from 61 to 84.4 tok/s (+38%). At C=32, the aggregate gain narrows to +11% (1,359 → 1,502 tok/s), and the per-request gain shrinks to just +6% (59 → 62.3 tok/s). The server is beginning to saturate.

At high concurrency (C=64–128), the tables turn dramatically. Aggregate throughput at C=64 drops from 2,327 to 1,571 tok/s (-32%), and at C=128 from 2,800 to 1,635 tok/s (-42%). The per-request throughput, however, continues to improve: from 44 to 65.7 tok/s at C=64 (+49%) and from 41 to 64.5 tok/s at C=128 (+57%). This is the key insight of the message: MTP improves per-request latency across the board but caps aggregate throughput at high concurrency.

The Reasoning: Why MTP Hurts at High Concurrency

The assistant immediately identifies the likely cause: "the max_running_requests=48 limit from speculative decoding caps concurrent requests, and the extra compute for the draft model becomes overhead when already saturated." This is a critical observation rooted in the architecture of SGLang's speculative decoding implementation.

When the assistant enabled MTP, SGLang automatically reset max_running_requests to 48 (as seen in the warning log from [msg 6358]: "Max running requests is reset to 48 for speculative decoding"). Without MTP, the server could handle up to the configured limit — likely 128 or higher — allowing it to batch more requests together and achieve higher aggregate throughput. With MTP, the server is artificially constrained to 48 concurrent requests, and each request requires additional computation for the draft model's forward pass. At high concurrency, where the GPU is already fully utilized processing the base model's forward passes, adding draft model computation creates overhead that reduces the total number of tokens the system can produce per second.

The per-request throughput improvement at high concurrency (+49% at C=64, +57% at C=128) reflects the fact that each individual request benefits from speculative decoding — it gets more tokens per step — but fewer requests can run simultaneously, so the total system throughput drops. This is a classic throughput-vs-latency tradeoff, and the assistant's analysis correctly identifies it.

The Assumptions and Their Implications

The assistant makes several implicit assumptions in this analysis. First, it assumes that the max_running_requests cap is the primary bottleneck. This is a reasonable hypothesis — the warning log explicitly mentions it — but it is not proven by the data. Other factors could contribute: the draft model's memory consumption (1.91 GB per rank) reduces the available KV cache space, potentially limiting batch sizes; the speculative verification step may introduce synchronization overheads that don't scale well; or the Mamba-specific scheduling strategy (extra_buffer) may interact poorly with high concurrency.

Second, the assistant assumes that the benchmark workload (1000-in/1000-out with fixed token IDs) is representative of the actual production workload. The assistant notes that "the per-request tok/s actually improves across all concurrency levels, which is the metric that matters more for the agentic coding use case." This is a value judgment: for an interactive coding assistant, users care about response latency, not aggregate server throughput. But this assumption would need validation against real traffic patterns.

Third, the assistant implicitly assumes that the --max-running-requests override would restore aggregate throughput at high concurrency. This is left as an open question — the assistant checks the logs but does not immediately implement the fix. The message ends with a bash command to verify the accept rate, suggesting the assistant is gathering more data before making a decision.

The Thinking Process: What the Message Reveals

The structure of message [msg 6366] reveals the assistant's analytical methodology. It begins with the raw data — the comparison table — which is presented with clear column headers and percentage changes. This is not a casual observation; it is a structured comparison that required the assistant to retrieve previous benchmark results (from before MTP was enabled) and align them with the new results.

The assistant then interprets the data, identifying the pattern ("MTP gives significant improvements at low to medium concurrency, but aggregate throughput drops at high concurrency"). This is followed by a causal hypothesis ("the max_running_requests=48 limit from speculative decoding caps concurrent requests"). Finally, the assistant acknowledges the nuance ("the per-request tok/s actually improves across all concurrency levels, which is the metric that matters more for the agentic coding use case") and proposes a next step (checking if --max-running-requests can be increased).

The bash command at the end — checking the accept rate from the logs — is a diagnostic step. The assistant wants to confirm that the speculation is still working correctly under load before deciding whether to increase the cap. This is disciplined engineering: verify the mechanism before tuning the parameter.

Input and Output Knowledge

To fully understand this message, the reader needs several pieces of input knowledge. They need to understand what speculative decoding is — the concept of a lightweight draft model predicting tokens that a base model verifies in parallel. They need to understand the difference between aggregate throughput (tokens per second across all concurrent requests) and per-request throughput (tokens per second for an individual request). They need to know that SGLang's speculative decoding implementation automatically caps max_running_requests at 48, and that this cap can be overridden. And they need to know the previous benchmark results (without MTP) to interpret the comparison.

The output knowledge created by this message is substantial. It provides a quantitative characterization of MTP's performance across the full concurrency spectrum for a 122B-parameter hybrid model on Blackwell GPUs. It establishes that MTP provides 12–45% per-request improvement at low concurrency but causes 32–42% aggregate throughput regression at high concurrency. It identifies the max_running_requests cap as the likely culprit. And it frames the tradeoff in terms of the actual use case: for agentic coding (latency-sensitive), MTP is beneficial; for high-throughput batch serving, it may not be.

The Broader Significance

This message is a microcosm of the challenges in deploying speculative decoding in production. The technique is not a universal win — it has a cost (extra GPU memory, extra computation, concurrency caps) that must be weighed against its benefits (higher per-request throughput, better token acceptance). The optimal configuration depends on the workload: a chatbot serving a few dozen simultaneous users will benefit from MTP, while a batch inference pipeline processing thousands of requests may not.

The assistant's response to this tension is measured and data-driven. It does not declare MTP a success or failure; it presents the data, identifies the tradeoff, and aligns the decision with the use case. This is the hallmark of good systems engineering: understanding that there are no silver bullets, only tradeoffs that must be measured and managed.

The message also demonstrates the importance of benchmarking at multiple concurrency levels. A single-request test would have shown MTP as a clear win (+23%). A high-concurrency test would have shown it as a clear loss (-42%). Only by testing across the full spectrum does the complete picture emerge — and that picture is more nuanced than either extreme suggests.

Conclusion

Message [msg 6366] is a masterclass in empirical performance analysis. It takes a complex technique (MTP speculative decoding), measures it across a realistic range of conditions, identifies the key tradeoff (per-request latency vs. aggregate throughput), and ties the decision to the actual use case. The assistant's reasoning is clear, its assumptions are explicit, and its next steps are well-motivated. For anyone deploying speculative decoding in production, this message captures the essential tension: speculation accelerates individual requests but can cap system throughput, and the right choice depends on whether you are optimizing for the user's experience or the server's utilization.