The Moment of Verification: Confirming MTP Speculation Delivers on Blackwell GPUs

In the long arc of deploying large language models on cutting-edge hardware, few moments are as tense as the first verification of a newly enabled optimization. Message [msg 6362] captures exactly such a moment: the assistant, having just enabled Multi-Token Prediction (MTP) speculative decoding on an SGLang deployment serving Qwen3.5-122B-A10B across four NVIDIA Blackwell RTX PRO 6000 GPUs, runs a quick log inspection to determine whether the optimization is actually working. The message is brief—a single bash command and its output—but it represents the culmination of an extensive debugging and configuration effort spanning driver troubleshooting, IOMMU architecture investigation, and speculative decoding research.

The Message in Full

The assistant writes:

Working correctly. Now let me run the benchmark to see if MTP gives a throughput boost:

>

``bash ssh root@10.1.230.174 'journalctl -u sglang-qwen --since "30 seconds ago" --no-pager 2>&1 | grep -E "throughput|spec_accept" | head -5' ``

>

`` Mar 09 16:03:25 llm-two sglang-qwen[15503]: [2026-03-09 16:03:25 TP0] Prefill batch, #new-seq: 1, #new-token: 22, #cached-token: 0, full token usage: 0.00, mamba usage: 0.02, #running-req: 0, #queue-req: 0, input throughput (token/s): 1.49, cuda graph: False Mar 09 16:03:26 llm-two sglang-qwen[15503]: [2026-03-09 16:03:26 TP0] Decode batch, #running-req: 1, #full token: 92, full token usage: 0.00, mamba num: 3, mamba usage: 0.02, accept len: 1.95, accept rate: 0.97, cuda graph: True, gen through... ``

The output is truncated—the head -5 limit and the log line ending with "gen through..." suggest the full throughput metric was cut off—but the critical data is visible. The assistant sees accept len: 1.95 and accept rate: 0.97, which are the key metrics for speculative decoding performance. An acceptance rate of 0.97 means 97% of the tokens predicted by the draft model (the MTP head) are accepted by the target model. An acceptance length of 1.95 means that on average, nearly two tokens are generated per speculation step. These numbers are exceptionally good and immediately validate the MTP optimization.

Why This Message Was Written: The Context of Verification

To understand why this message exists, one must appreciate the journey that led to it. The assistant had been working for hours—across multiple sessions and segments—to deploy and optimize an LLM serving stack on a novel hardware configuration. The system featured four NVIDIA Blackwell RTX PRO 6000 GPUs connected via PCIe, running on a Proxmox host with SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) enabled. This combination of cutting-edge GPU architecture and advanced virtualization features created a cascade of compatibility issues.

The most significant roadblock had been GPU P2P (Peer-to-Peer) DMA. In the immediately preceding messages ([msg 6334] through [msg 6348]), the assistant had attempted to enable P2P DMA by setting IOMMU identity domains for the NUMA0 GPUs. This approach—setting individual IOMMU groups to "identity" mode so that DMA translations bypass the IOMMU—had worked conceptually when tested manually. The assistant created a systemd service (gpu-iommu-identity.service) to apply the setting at boot time, before the NVIDIA driver loaded. However, a critical discovery emerged: the Blackwell GPU's FSP (Firmware Security Processor) requires specific DMA mappings set up by the kernel's DMA API in translation mode during its boot sequence. Identity mode breaks this initialization, causing the FSP to fail with error code 0x177. This meant that per-group IOMMU identity domains were fundamentally incompatible with Blackwell GPUs—the approach could not work regardless of timing.

Faced with this definitive dead end, the assistant pivoted. Rather than continuing to chase P2P DMA, it turned to an orthogonal optimization: MTP (Multi-Token Prediction) speculative decoding, also referred to in SGLang as NEXTN speculation. This technique uses a lightweight "draft" model—in Qwen3.5's case, a small MTP head attached to the main model—to predict multiple future tokens in a single forward pass. The main model then verifies these predictions, and accepted tokens are generated at reduced latency because multiple tokens can be produced per verification step.

The assistant had researched the MTP flags in [msg 6348], a subagent task that explored the SGLang codebase to identify the correct command-line parameters. The research revealed that for Qwen3.5 models, the flag --speculative-algorithm NEXTN was needed, which SGLang internally converts to the EAGLE algorithm path. The MTP draft model is loaded automatically from the model's own checkpoint. Additional flags like --speculative-eagle-max-draft 3 controlled the number of speculated tokens per step. The assistant then updated the systemd service file ([msg 6352]–[msg 6354]), deployed it ([msg 6355]), and restarted SGLang ([msg 6356]). After waiting for the server to become ready ([msg 6357]), it confirmed that the MTP draft model had loaded successfully ([msg 6358]–[msg 6359]), and ran a smoke test to verify the model was serving requests ([msg 6361]).

Message [msg 6362] is the next logical step: the first real verification that MTP is not just loaded, but actually delivering the promised throughput improvement. The assistant is not running a formal benchmark yet—that would come later—but is checking the server's internal metrics to see if speculative decoding is active and producing acceptable results.

The Thinking Process Visible in the Message

The assistant's reasoning in this message is concise but reveals several layers of decision-making. The opening phrase "Working correctly" refers to the smoke test in [msg 6361], which confirmed that the model returned the correct answer (345 for 15×23). But the assistant knows that a model serving correct answers does not guarantee that MTP speculation is actually engaged. The speculative decoding path could be silently falling back to standard autoregressive generation if the draft model or speculation logic failed to initialize properly.

The choice to check the logs via journalctl with a --since "30 seconds ago" filter is deliberate. The assistant wants to see the most recent activity—ideally, the decode batch from the smoke test request. By grepping for both "throughput" and "spec_accept", the assistant covers two bases: the general throughput metric (which would show tokens per second) and the specific speculative decoding acceptance metrics (which would confirm MTP is active). The head -5 limit is a practical choice to avoid log noise.

The output reveals exactly what the assistant needed to see. The decode batch log line includes accept len: 1.95, accept rate: 0.97. These numbers are the gold standard for speculative decoding health. An acceptance rate of 0.97 means the draft model is predicting tokens that the main model almost always agrees with—a sign that the MTP head is well-calibrated. An acceptance length of 1.95, given that the maximum speculation depth was set to 3 (via --speculative-eagle-max-draft 3), means the speculation is productive: on average, nearly two tokens are generated per speculation step instead of one. This roughly doubles the effective decode throughput for single requests, at least for the portion of generation where speculation is active.

The assistant also sees cuda graph: True in the decode log line, confirming that CUDA graph capture is working for the decode path—another optimization that reduces kernel launch overhead. The mamba num: 3 and mamba usage: 0.02 fields are relevant because Qwen3.5-122B-A10B is a hybrid Mamba/attention model; the Mamba layers are handled specially in the speculative decoding path.

Assumptions and Input Knowledge

This message rests on several assumptions, most of which are well-founded. The assistant assumes that the log metrics from a single smoke-test request are representative of general performance—a reasonable assumption for a first check, though formal benchmarking with multiple requests and varying concurrency would be needed for definitive numbers. The assistant also assumes that the accept len and accept rate metrics are accurate reflections of speculative decoding behavior, which depends on the SGLang codebase being correctly instrumented.

The input knowledge required to understand this message is substantial. One must know:

Output Knowledge Created

This message creates several pieces of actionable knowledge. First and foremost, it confirms that MTP speculation is working correctly on Blackwell GPUs with the Qwen3.5-122B-A10B model—a non-trivial validation given the earlier compatibility issues with flash-attn, FP4 backends, and CUDA versions. Second, it establishes a baseline acceptance rate (0.97) and acceptance length (1.95) that can be compared against future tuning attempts. Third, it implicitly validates that the SGLang nightly build (which had been patched for SM120 support) correctly implements the speculative decoding path for hybrid Mamba/attention models.

The truncated throughput metric ("gen through...") is a minor loss of information—the full line would have shown the exact tokens-per-second figure. However, the assistant can easily retrieve this with a follow-up command, and the acceptance metrics are arguably more important at this stage because they diagnose why throughput is what it is.

The Broader Significance

Message [msg 6362] represents a turning point in the session. After the IOMMU identity domain approach was definitively ruled out (Blackwell's FSP cannot tolerate identity mode), the assistant needed a win—a tangible optimization that would improve throughput without requiring P2P DMA. MTP speculation delivered that win. The acceptance rate of 0.97 is remarkably high, suggesting that the Qwen3.5 model's MTP head is exceptionally well-trained for this task. In many speculative decoding deployments, acceptance rates of 0.6–0.8 are considered good; 0.97 is outstanding.

This success also validates the assistant's decision-making process. When the P2P DMA path hit a fundamental hardware incompatibility, the assistant did not waste time trying workarounds (FLR, SBR, CXL bus resets were all considered and dismissed). Instead, it pivoted to an orthogonal optimization that did not depend on P2P at all. MTP speculation works entirely within each GPU's local memory—the draft model and target model share the same weights and KV cache—so it is unaffected by the NCCL P2P disablement that was required to work around the IOMMU corruption issue.

The message also demonstrates a disciplined approach to optimization: enable one change, verify it works, measure the impact, and only then move on. The assistant did not jump to a full benchmark suite immediately; it first checked the server logs for the basic health metrics. This incremental verification prevents wasted effort—if the acceptance rate had been 0.1, the assistant would have known immediately that something was wrong with the MTP configuration, rather than spending time running a full benchmark only to get confusing results.

Conclusion

Message [msg 6362] is a small but pivotal moment in a complex deployment story. In a few lines of log output, the assistant confirms that MTP speculative decoding is working on Blackwell GPUs with an exceptional 97% token acceptance rate and nearly 2 tokens generated per speculation step. This verification unlocks the next phase of optimization—benchmarking at various concurrency levels, tuning speculation depth, and potentially exploring the spec_v2 overlap path for further gains. The message exemplifies the scientific method applied to systems engineering: form a hypothesis (MTP will improve throughput), implement the change, and immediately verify the mechanism is functioning before measuring the outcome. It is a testament to the value of incremental, metrics-driven optimization in complex ML infrastructure deployments.