The Benchmark That Validated Everything: A Post-Mortem of IOMMU-Induced P2P Corruption on Blackwell GPUs

Introduction

In any large-scale machine learning deployment, the moment of truth arrives not when the server starts, but when the benchmark runs. Message <msg id=6231> captures precisely that moment in a months-long optimization journey across 8× NVIDIA RTX PRO 6000 Blackwell GPUs. After reconfiguring GPU topology across Proxmox containers and SEV-SNP virtual machines, diagnosing silent P2P DMA corruption caused by IOMMU translation, and applying a surgical workaround, the assistant finally runs a throughput benchmark that validates the entire stack. The numbers are striking: 108 tok/s single-stream, climbing to over 2,300 tok/s at moderate concurrency, with hints of even higher throughput at C=128. This message is the payoff for an intense debugging session — and it tells a rich story about systems integration, hardware limitations, and the art of working around platform constraints.

The Message in Full

The subject message reads:

[assistant] Let me update the benchmark script for the new model name and copy it over: [bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/bench_qwen.py root@10.1.230.174:/root/bench_qwen.py && ssh root@10.1.230.174 '~/ml-env/bin/python3 /root/bench_qwen.py 1 4 16 32 64 128' 2>&1 Warming up... Warmup: 100 tokens, ok=True

>

C | Agg tok/s | Per-req tok/s | Tokens | Wall(s) | Ok ----------------------------------------------------------------- 1 | 108.3 | 109.8 | 4000 | 36.9 | 4 4 | 286.2 | 81.6 | 7102 | 24.8 | 8 16 | 873.0 | 60.6 | 29036 | 33.3 | 32 32 | 1359.3 | 59.4 | 51026 | 37.5 | 64 64 | 2327.3 | 43.6 | 109760 | 47.2 | 128 128 ...

At first glance, this looks like a routine benchmark run. But the context transforms it into something far more significant: this is the first successful multi-GPU distributed inference run after the IOMMU fix.

Why This Message Was Written: The Debugging Arc

To understand why <msg id=6231> exists, we need to trace the crisis that preceded it. The server in question — an ASUS ESC8000A-E13 with dual AMD EPYC 9335 processors and 8× RTX PRO 6000 Blackwell GPUs — had been running with all 8 GPUs in a single LXC container for SGLang serving. Then a reconfiguration split the GPUs: 4 remained with the nvidia driver for the container, while the other 4 were moved to vfio-pci passthrough for a SEV-SNP confidential VM.

The SEV-SNP configuration required amd_iommu=on in full translation mode, unlike the previous iommu=pt (passthrough) mode. This seemingly minor change had catastrophic consequences: every GPU-to-GPU P2P DMA transfer produced corrupted data. The IOMMU, now actively translating all DMA addresses, was silently corrupting inter-GPU transfers. The symptom was a deadlocked NCCL init_torch_distributed — the distributed communication library would hang indefinitely because it received garbage data from P2P all-reduce operations.

The assistant diagnosed this through a chain of evidence: IO_PAGE_FAULTs in dmesg (AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x24000000000 flags=0x0030]), a CUDA P2P test showing every inter-GPU copy producing mismatches, and finally an NCCL test with NCCL_P2P_DISABLE=1 that worked perfectly. The fix was to force NCCL to use shared memory (SHM) transport instead of P2P DMA, applied via the system's sitecustomize.py and the systemd service file.

Message <msg id=6231> is the validation of that fix. Without it, we wouldn't know whether the workaround actually worked under realistic load — whether the SHM transport could handle the bandwidth demands of tensor-parallel inference across 4 GPUs.## How Decisions Were Made

The message itself is concise, but the decisions embedded in it are numerous. First, the assistant chose to update the benchmark script for the new model name — Qwen3.5-122B-A10B BF16, a 122-billion-parameter Mixture-of-Experts model with 10 billion active parameters per token. This model replaced the previous Qwen3.5-397B NVFP4, which was too large for the 4-GPU configuration after the topology split. The decision to benchmark with concurrency levels of 1, 4, 16, 32, 64, and 128 reflects a systematic approach: start with single-stream latency, then progressively increase load to find the saturation point. The benchmark script itself, bench_qwen.py, had been developed over previous sessions and was being reused with a simple model name update — a deliberate choice to maintain measurement consistency across model iterations.

The assistant also decided to run the benchmark remotely via SSH rather than locally, executing the Python script inside the container's ML environment (~/ml-env/bin/python3). This was necessary because the SGLang server ran on the remote container (IP 10.1.230.174), not on the local workstation. The benchmark script was copied over via scp before execution, ensuring the latest version was used.

Assumptions Made

Several assumptions underpin this message. The most critical is that NCCL_P2P_DISABLE=1 would not cause performance degradation severe enough to negate the benefits of multi-GPU tensor parallelism. This was a gamble: P2P DMA is the fastest path for inter-GPU communication on PCIe, and disabling it forces NCCL to fall back to shared memory (SHM) transport, which involves an extra copy through host memory. The benchmark results vindicate this assumption — 108 tok/s single-stream and 2,327 tok/s at C=64 are excellent numbers for a 122B BF16 model on 4 GPUs — but the assistant had no guarantee of this beforehand.

Another assumption was that the IOMMU corruption was the only issue preventing SGLang from starting. The assistant had already fixed a driver version mismatch (container userspace at 565 vs host kernel at 590) and updated the service file with the new environment variables. If there had been additional problems — model loading failures, CUDA version incompatibilities, or MoE routing bugs — the benchmark would have failed. The fact that it succeeded confirms the diagnosis was complete.

The assistant also assumed that the benchmark script's warmup phase (100 tokens) was sufficient to stabilize GPU clock speeds and memory bandwidth before measurement. This is a standard practice in ML benchmarking, but on Blackwell GPUs with dynamic voltage/frequency scaling, a longer warmup might have been necessary. The results suggest the assumption was reasonable.

Input Knowledge Required

To fully understand this message, one needs considerable context. The reader must know that NCCL_P2P_DISABLE=1 is an environment variable that forces NVIDIA's Collective Communications Library to avoid direct GPU-to-GPU peer access, instead routing data through the CPU's shared memory. They must understand that IOMMU (I/O Memory Management Unit) in full translation mode remaps all DMA addresses, and that on AMD platforms with SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging), the IOMMU must operate in this mode for security guarantees.

The reader also needs to know the hardware topology: 4× RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 x16, with no NVLink or NVSwitch between them. This makes P2P DMA the primary bottleneck for tensor-parallel communication, and any degradation of that path directly impacts throughput. The model details matter too: Qwen3.5-122B-A10B is a Mixture-of-Experts architecture with 122B total parameters but only 10B active per token, meaning it fits in ~60 GB of GPU memory per card (BF16) while leaving room for KV cache.

The benchmark methodology itself requires interpretation. The "Agg tok/s" column shows aggregate throughput across all concurrent requests, while "Per-req tok/s" divides that by the concurrency level. The "Tokens" column shows total tokens generated across all requests, "Wall(s)" is elapsed wall-clock time, and "Ok" is the number of completed requests. The truncation at C=128 (the line ends with "128 ...") suggests the benchmark was still running or the output was cut off, but the trend is clear: throughput increases with concurrency up to at least C=64.## Output Knowledge Created

This message creates several important pieces of knowledge. First, it establishes a baseline throughput for Qwen3.5-122B-A10B BF16 on 4× Blackwell GPUs with NCCL P2P disabled — a configuration that had never been tested before. The numbers (108 tok/s single-stream, 2,327 tok/s at C=64) serve as a reference point for future optimization work. Any future change — whether upgrading CUDA, enabling P2P via BIOS fixes, or switching to a different communication protocol — can be measured against this baseline.

Second, the benchmark validates the IOMMU workaround. Before this message, the assistant had only proven that NCCL all-reduce worked correctly in a toy test with 4 GPUs. The benchmark proves that the workaround holds up under realistic inference load, with actual model weights and attention computations happening in parallel. This is a non-trivial validation: tensor-parallel inference requires frequent all-reduce operations for every transformer layer, and any communication bottleneck would show up immediately in the throughput numbers.

Third, the message implicitly documents the scaling behavior of this model on this hardware. The per-request throughput drops from 109.8 tok/s at C=1 to 43.6 tok/s at C=64, which is typical for throughput-oriented serving (more requests means more queueing and batching overhead). The aggregate throughput scales almost linearly from C=1 to C=16 (108 → 873 tok/s), then continues growing but with diminishing returns. This data is valuable for capacity planning: if a production workload requires 2,000 tok/s aggregate, the system can handle it at around C=64 with room to spare.

Mistakes and Incorrect Assumptions

The most notable limitation of this message is what it doesn't show: the C=128 result is truncated. The benchmark line ends with "128 ...", suggesting either the output was cut off by the tool's timeout or the benchmark was still running when the message was captured. This is a minor loss — the C=128 result would have shown whether throughput continued to scale or plateaued — but it doesn't invalidate the data we have.

A more subtle issue is that the benchmark measures output token generation only, not time-to-first-token (TTFT) or end-to-end latency for a complete request-response cycle. In production serving, TTFT is often as important as generation throughput, especially for interactive applications. The benchmark also doesn't test with variable-length prompts or the model's thinking/reasoning capabilities, which could introduce additional overhead.

The assistant also assumed that disabling P2P would not cause correctness issues beyond performance. While the NCCL test confirmed all-reduce correctness, there could be subtle numerical differences in the model's output due to different communication ordering in SHM transport vs P2P DMA. The smoke test (asking "What is 7 * 13?") returned the correct answer "91", but this is hardly exhaustive validation.

The Thinking Process Visible in the Message

The message itself is terse, but the thinking process is visible in its structure. The assistant begins by stating the intent ("Let me update the benchmark script for the new model name and copy it over"), then executes two commands: a file copy and a remote benchmark invocation. The choice of concurrency levels — 1, 4, 16, 32, 64, 128 — reveals a systematic doubling strategy designed to find the inflection point where throughput stops scaling linearly.

The warmup phase (100 tokens) shows that the assistant understands the importance of stabilizing GPU state before measurement. The benchmark output format — with columns for concurrency, aggregate throughput, per-request throughput, total tokens, wall time, and completion count — reflects a carefully designed measurement framework built over previous sessions. The fact that all requests completed successfully ("Ok" column matches concurrency × 2 for most rows) confirms that the server is stable under load.

The truncation at C=128 is itself informative: it suggests the benchmark was still running when the SSH command timed out or the output was captured. The assistant likely intended to show the full results in a follow-up message, but the conversation capture stopped here. This is a reminder of the asynchronous nature of these sessions — the assistant issues commands and waits for results, but tool outputs can be truncated by timeouts or size limits.

Conclusion

Message <msg id=6231> is a quiet victory lap after an intense debugging session. It doesn't announce "the fix works" in dramatic terms — it simply runs a benchmark and lets the numbers speak. The 108 tok/s single-stream and 2,327 tok/s at C=64 are the evidence that the IOMMU workaround is production-viable. For anyone who followed the debugging arc — the IO_PAGE_FAULTs, the corrupted P2P transfers, the driver mismatches, the service file edits — this message is the payoff. It transforms a hypothesis ("NCCL_P2P_DISABLE=1 should work") into validated knowledge, and it establishes a baseline that will guide every future optimization on this machine.