The MTP Benchmark: When Speculation Accelerates — and When It Doesn't

In the high-stakes world of large language model serving, every token-per-second counts. When the assistant in this opencode session enabled Multi-Token Prediction (MTP) speculation on a SGLang deployment serving Qwen3.5-122B-A10B, the initial signs were encouraging: a single-request throughput of 135 tok/s, up from roughly 110 tok/s without speculation — a ~23% improvement. But a single data point tells only part of the story. Real production deployments must handle concurrent requests, and the interaction between speculative decoding and concurrency is complex. Message [msg 6365] captures the moment the assistant ran the comprehensive benchmark to find out exactly where MTP helps and where it might hurt.

The Message: A Benchmark in Flight

The message itself is deceptively simple — a single bash command executed over SSH on the remote inference server:

[assistant] [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 /root/bench_qwen.py 1 4 16 32 64 128' 2>&1
Warming up...
  Warmup: 100 tokens, ok=True

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      121.5 |          123.1 |     4000 |     32.9 |    4
   4 |      400.5 |          118.6 |     7167 |     17.9 |    8
  16 |     1209.4 |           84.4 |    29417 |     24.3 |   32
  32 |     1502.4 |           62.3 |    51721 |     34.4 |   64
  64 |     1571.1 |           65.7 |    97000 |     61.7 |  128
 128 ...

The benchmark script bench_qwen.py (copied to the server just moments earlier in [msg 6364]) sends requests with a fixed 1000-token input and requests 1000 output tokens — a methodology designed to measure raw throughput without the variability of real-time generation. The concurrency levels tested — 1, 4, 16, 32, 64, 128 — span the range from single-user interactive use to high-throughput batch serving. The output is clean and tabular, showing aggregate throughput, per-request throughput, total tokens generated, wall-clock time, and success count for each concurrency level.

But the output is truncated at "128 ...", hinting that the benchmark may still have been running or its output was clipped. The full picture emerges in the very next message ([msg 6366]), where the assistant compares these results against the pre-MTP baseline.

The Reasoning Behind the Benchmark

This message did not appear in isolation. It is the culmination of a deliberate, multi-step process spanning dozens of earlier messages. The assistant had spent the preceding session (Segment 40) diagnosing and fixing a severe P2P DMA corruption issue under SEV-SNP IOMMU, ultimately disabling NCCL P2P with NCCL_P2P_DISABLE=1 to achieve stability. With the system stabilized, the assistant pivoted to optimization — specifically, enabling MTP speculation.

The chain of reasoning is visible in the messages leading up to [msg 6365]:

  1. [msg 6347]: The assistant recognized that MTP speculation was the next optimization to pursue, creating a todo item and launching a research sub-task.
  2. [msg 6348]: A sub-agent explored the SGLang codebase to understand MTP/NEXTN flags for hybrid Mamba/attention models like Qwen3.5-122B-A10B.
  3. [msg 6349]: Armed with research findings, the assistant stopped the SGLang service.
  4. <msg id=6351-6354>: The service file was edited to add --speculative-algorithm NEXTN and related flags.
  5. <msg id=6355-6359>: The updated service was deployed, SGLang restarted, and the assistant verified that MTP was active by inspecting logs — confirming speculative_algorithm=&#39;EAGLE&#39; (NEXTN converted to EAGLE internally), the draft model loaded at 1.91 GB per rank, and overlap scheduling enabled via SPEC_V2.
  6. [msg 6363]: A smoke test showed MTP working with accept len: 2.00 and accept rate: 1.00 — a perfect 100% acceptance rate on a simple query, yielding 135 tok/s. The benchmark in [msg 6365] was the natural next step: quantify the improvement systematically across the full concurrency range. The assistant needed to know not just that MTP works, but how well it works under different load conditions. This is the difference between a demo and a production deployment.

What the Results Reveal

The benchmark results tell a nuanced story. At low concurrency (C=1), MTP delivers 121.5 aggregate tok/s — a solid improvement over the baseline of 108 tok/s. At C=4, the improvement is dramatic: 400.5 tok/s versus 286 tok/s, a 40% boost. At C=16, the pattern continues with 1209.4 tok/s versus 873 tok/s (+38%).

But then something interesting happens. At C=32, the improvement narrows to just 11% (1502 vs 1359). At C=64, MTP actually underperforms the baseline: 1571 tok/s versus 2327 tok/s, a 32% regression. At C=128, the gap widens further: approximately 1635 tok/s versus 2800 tok/s, a 42% drop.

This is not a bug — it is a predictable consequence of speculative decoding's resource demands. The draft model consumes 1.91 GB of GPU memory per rank, reducing the available memory for the KV cache and thereby limiting the maximum number of concurrent requests that can be served. SGLang's own warning (seen in [msg 6358]) confirms: "Max running requests is reset to 48 for speculative decoding." With a hard cap of 48 concurrent requests, sending 64 or 128 requests simultaneously creates queueing pressure that negates the per-request throughput gains.

The per-request tok/s metric tells the other side of the story. Even at C=128, per-request throughput improves from 41 tok/s (baseline) to 64.5 tok/s (MTP) — a 57% improvement in individual response latency. For the agentic coding use case that motivated this entire deployment, where individual users wait for responses, this per-request improvement may matter more than aggregate throughput.

Assumptions and Their Validity

The assistant made several assumptions when running this benchmark. First, it assumed that the benchmark script, designed for the earlier NVFP4 model variant, would work correctly for the BF16 model now being served. The script's comment — "Quick throughput benchmark for Qwen3.5 NVFP4" — suggests it was written for a different quantization format. However, the benchmark methodology (fixed token IDs, HTTP requests to the SGLang API) is model-agnostic, so this assumption was safe.

Second, the assistant assumed the server was fully warmed up and stable. The warmup round completed successfully ("Warmup: 100 tokens, ok=True"), confirming that the model weights were loaded and CUDA graphs were compiled. However, the first few requests at each concurrency level may still have been warming the CUDA graph cache, potentially biasing the results slightly downward.

Third, the assistant implicitly assumed that the comparison against the "No MTP" baseline (performed in [msg 6366]) was valid. The baseline results were collected in a previous session with the same model, same hardware, and same benchmark script — but with different SGLang build and configuration. If any other system parameters changed between runs (e.g., CUDA version, kernel driver, memory configuration), the comparison would be imperfect. The assistant appears to have been aware of this, as the comparison table in [msg 6366] is presented as a direct "before/after" without caveats.

The Deeper Significance

Message [msg 6365] represents a critical decision point in the optimization journey. The benchmark results reveal that MTP is not a universal win — it excels at low-to-medium concurrency but becomes a liability under high load. This insight directly shapes the deployment strategy: for the primary use case of agentic coding (typically 1-4 concurrent users), MTP provides a 12-45% improvement. For batch processing at high concurrency, the assistant would need to either disable MTP dynamically or increase the --max-running-requests limit beyond the default 48.

The truncated "128 ..." line is almost poetic — it hints at unfinished business. The assistant would need to investigate whether increasing max-running-requests could recover the lost aggregate throughput at high concurrency, or whether the fundamental memory overhead of the draft model makes high-concurrency MTP impractical on this hardware configuration.

In the broader context of the session, this benchmark marks the transition from "does it work?" to "how well does it work under real conditions?" — the essential shift from engineering prototype to production deployment. The assistant's systematic approach — research, implement, verify, benchmark, analyze — is a model of disciplined systems engineering, and message [msg 6365] is the moment where the numbers speak for themselves.