Verifying MTP Speculation: The Moment Theory Meets Reality in a Blackwell LLM Deployment

Introduction

In the lifecycle of deploying a large language model (LLM) serving stack, there is a critical moment that separates theoretical optimization from actual working throughput: the first time you start the service with a new configuration and watch the logs to see if the changes actually take effect. Message [msg 6356] captures exactly this moment. After an extensive journey through IOMMU configuration, P2P DMA debugging, Blackwell firmware incompatibilities, and SGLang source code research, the assistant starts the SGLang service with Multi-Token Prediction (MTP) speculation enabled and checks the journal logs to confirm it is working. The result is a quiet but deeply significant log line: #speculation: 5, #accepted: 5, #draft: 5, spec_accept: 1.00.

The Message

The assistant executes a single bash command that starts the SGLang service and retrieves the first 20 lines of the systemd journal:

[assistant] [bash] ssh root@10.1.230.174 'systemctl start sglang-qwen.service && echo "Started" && sleep 5 && journalctl -u sglang-qwen -n 20 --no-pager 2>&1'
Started
Mar 09 15:54:13 llm-two sglang-qwen[14363]: [2026-03-09 15:54:13] INFO:     127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Mar 09 15:54:24 llm-two sglang-qwen[14403]: [2026-03-09 15:54:24 TP0] Prefill batch, #new-seq: 1, #new-token: 22, #cached-token: 0, full token usage: 0.00, mamba usage: 0.01, #running-req: 0, #queue-req: 0, input throughput (token/s): 1.82, cuda graph: False
Mar 09 15:54:24 llm-two sglang-qwen[14403]: [2026-03-09 15:54:24 TP0] Decode batch, #running-req: 1, #token: 1, #speculation: 5, #accepted: 5, #draft: 5, spec_accept: 1.00, new token usage: 0.00, mamba usage: 0.01, #running-req: 1, #queue-req: 0, input throughput (token/s): 0.09, output throughput (token/s): 0.09, cuda graph: False

There are no secrets or credentials to redact in this message — it is purely operational output from a production LLM server.

Why This Message Was Written

The message exists because the assistant had just completed a multi-step process to enable MTP speculation on the Qwen3.5-122B-A10B model running across 4 NVIDIA RTX PRO 6000 Blackwell GPUs. The preceding messages in the conversation reveal the full chain of reasoning.

First, the assistant had spent considerable effort trying to restore P2P DMA between GPUs. The Blackwell GPUs are split across two NUMA domains on an AMD EPYC host. The NUMA0 GPUs (01:00.0, 11:00.0, 61:00.0, 71:00.0) are bound to the nvidia driver for serving, while the NUMA1 GPUs are on vfio-pci for an SEV-SNP virtual machine. The assistant attempted to set IOMMU identity domains for the NUMA0 GPU groups to enable direct P2P DMA, bypassing the IOMMU translation that was causing IO_PAGE_FAULT errors. However, this approach was definitively blocked by a critical discovery: the Blackwell GPU's Firmware Security Processor (FSP) fails to boot with error code 0x177 when IOMMU is in identity mode. The FSP requires specific DMA mappings set up by the kernel's DMA API in translation mode, and identity mode breaks this initialization. This was a fundamental hardware incompatibility — no amount of driver rebinding or PCI resets could clear it.

With P2P DMA restoration blocked, the assistant pivoted to the next major optimization: MTP speculation. This technique, also known as NEXTN speculation in SGLang's terminology, allows the model to predict multiple future tokens in a single forward pass, dramatically increasing throughput. The assistant had previously dispatched a subagent task ([msg 6348]) to research the SGLang codebase and determine the correct flags for enabling MTP on hybrid Mamba/attention models like Qwen3.5-122B-A10B. The research returned detailed findings about the --speculative-algorithm NEXTN flag, the --speculative-num-steps parameter, and the --speculative-eagle-path option.

Based on this research, the assistant edited the SGLang service file ([msg 6352] and [msg 6353]) to add the MTP flags. The updated service file was then deployed to the container ([msg 6355]). Message [msg 6356] is the verification step — the moment the assistant confirms that the configuration change actually produces the desired behavior.

How Decisions Were Made

The decision to use --speculative-algorithm NEXTN rather than EAGLE or EAGLE3 was informed by the subagent research. The Qwen3.5-122B-A10B model is a hybrid architecture combining Mamba state-space model layers with traditional attention layers. EAGLE and EAGLE3 speculation require separate draft model heads that are trained specifically for speculative decoding. NEXTN speculation, on the other hand, uses the model's own Multi-Token Prediction (MTP) heads — output heads that predict multiple future tokens simultaneously, which are part of the model's native architecture. For the Qwen3.5 family, the MTP heads are already present in the model weights, making NEXTN the natural choice.

The assistant also had to decide on the number of speculative steps. The log output shows #speculation: 5, meaning 5 speculative tokens are being generated per step. This value was set via --speculative-num-steps 5 in the service file. The choice of 5 is a common default that balances the increased computational cost of predicting multiple tokens against the throughput gains from speculation. Too few steps limits the potential speedup; too many steps increases the risk of rejection cascades where a single wrong prediction invalidates all subsequent speculative tokens.

The decision to use NCCL_P2P_DISABLE=1 (visible in the environment from earlier messages) was a pragmatic acceptance that P2P DMA between Blackwell GPUs under IOMMU translation mode is fundamentally broken. The nvidia driver's DmaRemapPeerMmio=1 parameter was already enabled and partially working — some GPU pairs could transfer data correctly while others could not — but the directional IOMMU mapping issue was inconsistent and unreliable. Rather than fighting a losing battle, the assistant chose to disable P2P entirely and rely on shared memory (SHM) transport for NCCL communication. This is a performance trade-off: SHM is slower than direct GPU P2P for cross-GPU transfers, but it is stable and predictable.

Assumptions Made

The assistant made several assumptions in this message. The most critical assumption was that the MTP configuration would work correctly with the hybrid Mamba/attention architecture. While the subagent research confirmed that NEXTN speculation is supported for hybrid models, the actual behavior at runtime depends on the specific model implementation in SGLang. The log line showing spec_accept: 1.00 (100% acceptance rate) on the first decode batch is encouraging, but it is based on a single request — a more rigorous benchmark would be needed to confirm sustained performance.

Another assumption was that the service file edits were syntactically correct and would be parsed properly by SGLang. The assistant edited the service file twice ([msg 6352] and [msg 6353]), and the final version was read back ([msg 6354]) before deployment. However, the read-back only showed the first 13 lines of the file (truncated with ...), so the assistant was operating with incomplete visibility into the final state of the service file.

The assistant also assumed that the previous SGLang instance was fully stopped before starting the new one. The log output shows two PIDs: 14363 (the old instance) still handling a POST request at 15:54:13, and 14403 (the new instance) processing prefills and decodes at 15:54:24. This overlap is normal during a service restart — the old process finishes its in-flight requests while the new process starts accepting new ones — but it does indicate that the shutdown sequence in [msg 6349] and [msg 6350] may not have been completely clean.

Input Knowledge Required

To understand this message, one needs knowledge of several domains. First, the SGLang serving framework and its speculative decoding architecture: what MTP/NEXTN speculation is, how it differs from EAGLE-style speculation, and what the log fields (#speculation, #accepted, #draft, spec_accept) represent. Second, the systemd service management model: how service files are structured, how systemctl start and journalctl work, and how service restarts interact with running processes. Third, the NVIDIA GPU ecosystem on Linux: CUDA, NCCL, the nvidia driver, and the IOMMU interaction model. Fourth, the specific hardware platform: AMD EPYC with SEV-SNP, Blackwell GPU architecture, and the Proxmox virtualization layer.

The assistant also needed knowledge of the Qwen3.5-122B-A10B model architecture — specifically that it is a hybrid Mamba/attention model with native MTP heads — to make the correct choice of speculative algorithm. This knowledge was acquired through the subagent research task in [msg 6348], which explored the SGLang source code to find the relevant flags and their compatibility with different model types.

Output Knowledge Created

This message created several pieces of output knowledge. The most immediate is the confirmation that MTP speculation is operational on the Qwen3.5-122B-A10B model with 4 Blackwell GPUs. The log line #speculation: 5, #accepted: 5, #draft: 5, spec_accept: 1.00 demonstrates that the speculative decoding pipeline is functioning correctly — the draft tokens are being generated, verified against the model's own predictions, and accepted at a perfect rate (for this single initial request).

The message also confirms that the service file deployment mechanism works: the updated service file was copied to the container, systemd was reloaded, and the service started without errors. This validates the operational workflow for future configuration changes.

Perhaps most importantly, the message establishes a new baseline for performance optimization. With P2P DMA definitively blocked by the Blackwell FSP/IOMMU incompatibility, MTP speculation becomes the primary lever for improving throughput. The assistant can now proceed to benchmark the speculation performance at various concurrency levels, tune the number of speculative steps, and potentially implement dynamic speculation disabling based on queue depth — all building on the foundation confirmed in this message.

Mistakes and Incorrect Assumptions

The most notable potential issue is the perfect 1.00 acceptance rate on the first decode batch. While this sounds ideal, it is almost certainly an artifact of the specific test conditions — a single request with a short prompt and a small number of generated tokens. In real-world usage with diverse prompts and longer generation sequences, the acceptance rate will vary based on how predictable the model's output is. A perfect acceptance rate on the first request should be interpreted as "the mechanism works" rather than "the mechanism is optimally tuned."

The assistant also did not verify that the --speculative-algorithm NEXTN flag was actually parsed correctly by SGLang. The log output does not explicitly state which speculative algorithm is in use — it only shows speculation-related fields in the decode batch log line. If there were a fallback mechanism or a silent failure in parsing the flag, the speculation might be using a different algorithm than intended. A more thorough verification would involve checking the SGLang startup logs for a line confirming the speculative algorithm configuration.

The Thinking Process Visible in the Message

The structure of the command itself reveals the assistant's thinking process. The command is a pipeline: start the service, echo "Started" to confirm the systemctl command succeeded, sleep 5 seconds to allow the service to initialize and process its first request, then dump the journal logs. This sequential approach shows that the assistant is thinking about timing — it knows that SGLang takes some time to load the model weights, compile CUDA graphs, and begin processing requests. The 5-second sleep is a heuristic guess at how long this initialization takes.

The choice of -n 20 (20 lines of journal output) is also revealing. The assistant wants enough context to understand the service state but not so much that the output is overwhelming. The resulting log output shows exactly the right information: the old instance finishing a request, the new instance starting prefills, and the critical decode batch line with speculation metrics.

The fact that the assistant checks the journal immediately after starting the service, rather than waiting for a benchmark or a more comprehensive test, shows that the primary goal at this moment is verification of the configuration, not performance measurement. The question being asked is "does MTP speculation work?" rather than "how much does MTP speculation improve throughput?" The latter question will come later, in subsequent messages.

Conclusion

Message [msg 6356] is a quiet but pivotal moment in a complex LLM deployment journey. After navigating IOMMU incompatibilities, Blackwell firmware limitations, and SGLang configuration research, the assistant confirms that MTP speculation is operational. The log line showing 5 speculative tokens accepted at a perfect rate is the first evidence that the theoretical optimization is delivering real results. This message bridges the gap between configuration and verification, between research and operation, and between the assistant's intent and the system's behavior. It is the moment the plan becomes reality.