The IOMMU Wall: Diagnosing and Defeating P2P DMA Corruption Under SEV-SNP

In the high-stakes world of large-scale ML inference deployment, the difference between a working system and a mysteriously hanging one often comes down to a single environment variable. Message [msg 6235] in this opencode session is a concise "deployment complete" summary that belies the extraordinary diagnostic effort required to get there. Written by the AI assistant after successfully launching the Qwen3.5-122B-A10B BF16 model across 4 NVIDIA RTX PRO 6000 Blackwell GPUs, this message serves as both a victory lap and a critical piece of systems documentation. It reports benchmark results reaching 2,800 tokens per second at high concurrency, but more importantly, it crystallizes the root cause of a days-long debugging saga: the collision between AMD SEV-SNP confidential computing and GPU peer-to-peer DMA.

The Message

The assistant writes:

## Deployment Complete: Qwen3.5-122B-A10B BF16 on TP=4

### Benchmark Results (1000-in / 1000-out, synthetic tokens)

| C | Agg tok/s | Per-req tok/s | |---|---|---| | 1 | 108 | 110 | | 4 | 286 | 82 | | 16 | 873 | 61 | | 32 | 1,359 | 59 | | 64 | 2,327 | 44 | | 128 | 2,800 | 41 |

### Key Details - Model: Qwen3.5-122B-A10B BF16 (125B total, 10B active, 256 experts) - Location: /shared/models/Qwen3.5-122B-A10B (234 GB) - KV cache: BF16, 780,820 tokens (~3x 262K context) - No MTP — disabled for now since it requires extra_buffer scheduler + spec_v2 (can add later) - No quantization — native BF16

### Critical Fix: IOMMU / P2P The other agent's SEV-SNP setup changed IOMMU from passthrough to full translation mode (amd_iommu=on without iommu=pt). This breaks GPU-to-GPU P2P DMA — transfers complete but with corrupted data (IO_PAGE_FAULTs in dmesg). Fixed by: - NCCL_P2P_DISABLE=1 in both sitecustomize.py and the service file - Removed NCCL_P2P_LEVEL=SYS from sitecustomize - NCCL falls back to SHM (shared memory) transport which works correctly

### Comparison to Qwen3.5-397B NVFP4 (8 GPUs)

| Metric | 397B NVFP4 (TP=8) | 122B BF16 (TP=4) | |---|---|---| | Single-stream tok/s | 172 | 108 | | C=32 agg tok/s | 2,156 | 1,359 | | KV cache tokens | 471K | 781K | | Model quality | Lower (FP4 quant) | Higher (native BF16, same architecture) |

The 122B is slower per-token (expected — BF16 compute vs FP4, and half the GPUs), but has ~66% more KV cache and is a native-precision model with no quantization artifacts.

This is a remarkably dense message. In just a few paragraphs, it reports performance data, documents a critical system-level workaround, explains tradeoffs between two model deployments, and provides enough context for someone familiar with the infrastructure to understand what happened. But to fully appreciate what this message represents, we need to understand the journey that produced it.

The SEV-SNP IOMMU Crisis

The story begins with a topology change. The Proxmox host running this ML infrastructure has 8 NVIDIA RTX PRO 6000 Blackwell GPUs. In a previous configuration, all 8 were available to a single LXC container running SGLang with tensor parallelism (TP) of 8. But a new requirement emerged: the system needed to support AMD SEV-SNP (Secure Encrypted Virtualization — Secure Nested Pages) confidential VMs. This required splitting the GPUs: 4 remained bound to the NVIDIA driver for the LXC container, while the other 4 were passed through to a VM via vfio-pci.

The critical, unanticipated consequence of this change was that SEV-SNP requires the IOMMU (I/O Memory Management Unit) to operate in full translation mode (amd_iommu=on without iommu=pt). Previously, the IOMMU was likely in passthrough mode (iommu=pt), which allows devices to DMA directly to physical addresses without translation. Full translation mode intercepts every DMA transaction, translating device addresses to physical addresses through IOMMU page tables. This is essential for SEV-SNP's memory encryption and isolation guarantees, but it has a devastating side effect for GPU workloads.

The Diagnostic Trail

The first symptom was that SGLang simply would not start. The NCCL (NVIDIA Collective Communications Library) distributed initialization would hang during init_torch_distributed, with no clear error message. The assistant began methodically working through the diagnostic chain.

The first clue came from kernel messages. Running dmesg | grep AMD-Vi revealed IO_PAGE_FAULT events — the IOMMU was logging page faults for DMA transactions originating from the GPUs. The fault addresses (like 0x24000000000 and 0x34000010000) were immediately recognizable as GPU BAR (Base Address Register) addresses of other GPUs. The GPUs were trying to perform peer-to-peer DMA directly to each other's memory, and the IOMMU was blocking these transactions.

But the situation was more insidious than a simple block. When the assistant tested P2P data transfers using PyTorch (see [msg 6216]), it discovered that torch.cuda.can_device_access_peer() returned True for every GPU pair — the driver believed P2P was available. Yet every actual data transfer produced corrupted output. Every single GPU-to-GPU copy across all 12 pairs resulted in "MISMATCH." The IOMMU was not cleanly blocking P2P; it was allowing the transfers to complete but with silently corrupted data. This is the worst possible failure mode: the system reports success but delivers garbage.

This silent corruption explained why NCCL hung. NCCL's P2P all-reduce algorithm would initiate transfers, receive corrupted data, and then either deadlock waiting for valid data or produce incorrect results that caused the distributed initialization logic to enter an unrecoverable state.

The Fix: NCCL_P2P_DISABLE=1

The solution, once the root cause was understood, was elegantly simple. By setting NCCL_P2P_DISABLE=1, the assistant forced NCCL to abandon P2P transport entirely and fall back to SHM (shared memory) transport. In SHM mode, GPU data is copied through host memory via the CPU, which goes through the IOMMU correctly because the CPU's memory controller is properly integrated with the IOMMU translation tables.

The assistant verified this fix by writing a dedicated NCCL test script (<msg id=6218-6219>) that ran a distributed all-reduce across 4 GPUs with NCCL_P2P_DISABLE=1. All ranks reported correct results: all_reduce OK (got 10.0, expected 10.0). The fix was then made permanent by updating /usr/lib/python3.12/sitecustomize.py and the systemd service file to set this environment variable automatically (see <msg id=6221, 6223>).

The assistant also removed NCCL_P2P_LEVEL=SYS from the sitecustomize, which had been an earlier attempt to limit P2P to system-level (same node) transfers. With full IOMMU translation, even system-level P2P was broken, so complete disablement was necessary.

Benchmark Results: What the Numbers Tell Us

The benchmark results in the message reveal a system that scales beautifully with concurrency. At a single concurrent request (C=1), the system delivers 108 tok/s aggregate (110 per-request). At C=128, aggregate throughput reaches 2,800 tok/s — a 26x improvement from single-stream, demonstrating excellent scheduler efficiency and memory bandwidth utilization.

The per-request throughput drops from 110 tok/s at C=1 to 41 tok/s at C=128, which is expected behavior: as more requests compete for the same GPU compute and memory bandwidth, each individual request slows down, but the system as a whole processes far more tokens per second. The relatively gentle decay (only 63% reduction in per-request speed at 128x concurrency) suggests the KV cache and scheduler are well-tuned.

The KV cache capacity of 780,820 tokens is impressive. This is roughly equivalent to 3 simultaneous 262K-token contexts or about 24 concurrent 32K-token sessions. The BF16 KV cache format (rather than FP8) was a deliberate choice: earlier in the session, the assistant had discovered that FP8 KV cache produced accuracy issues on the SM120 Blackwell architecture and forced BF16 to maintain model quality.

The Comparison Table: A Study in Tradeoffs

The comparison between the previous 397B NVFP4 deployment (on 8 GPUs) and the current 122B BF16 deployment (on 4 GPUs) is illuminating. The 397B model was faster in raw throughput — 172 tok/s single-stream versus 108 tok/s — but this comparison is apples-to-oranges in several dimensions.

First, the 397B model used NVFP4 quantization, which packs model weights into 4-bit floating point. This dramatically reduces memory footprint and increases arithmetic throughput on Blackwell's FP4 tensor cores, but at a cost to model quality. The 122B model runs in native BF16, preserving full precision. As the assistant notes, the 122B "is a native-precision model with no quantization artifacts."

Second, the 397B ran on 8 GPUs with TP=8, while the 122B runs on 4 GPUs with TP=4. Half the GPUs means half the compute and memory bandwidth, which directly explains the lower single-stream throughput. But the 122B has 66% more KV cache capacity (781K vs 471K tokens), which is critical for long-context applications.

Third, both models share the same Qwen3.5 architecture — the 122B is simply a smaller variant (125B total parameters, 10B active per token via Mixture of Experts with 256 experts). This means the deployment infrastructure and serving stack are identical, making the comparison meaningful despite the model size difference.

Deeper Implications: IOMMU, Confidential Computing, and GPU P2P

The IOMMU/P2P issue documented in this message has implications far beyond this single deployment. As confidential computing technologies like AMD SEV-SNP and Intel TDX become more prevalent, the tension between IOMMU translation and GPU P2P DMA will become a recurring challenge for ML infrastructure engineers.

GPU P2P DMA is essential for high-performance distributed training and inference. When GPUs can directly read and write each other's memory without CPU involvement, collective operations like all-reduce can achieve dramatically lower latency and higher bandwidth. NCCL's P2P transport is the default path for this reason. But P2P DMA bypasses the IOMMU — GPU devices address each other's BAR spaces directly, which is incompatible with IOMMU translation.

The assistant's investigation didn't stop at the software fix. In the messages following the deployment summary (<msg id=6236+), the assistant began exploring BIOS-level options to re-enable P2P DMA under IOMMU translation, investigating PCIe ACS (Access Control Services), ARI (Alternative Routing-ID Interpretation), and IOMMU coverage settings. This is the right approach: NCCL_P2P_DISABLE=1 is a workaround, not a solution. It forces data through host memory, adding latency and consuming CPU and memory bandwidth. A proper solution would involve configuring the IOMMU to allow specific P2P translations or using hardware features like PCIe ACS to enable direct device-to-device transfers within the same IOMMU group.

Why This Message Matters

Message [msg 6235] is a masterclass in technical communication under pressure. It was written at the end of a grueling diagnostic session that spanned GPU topology reconfiguration, kernel message analysis, CUDA P2P testing, NCCL debugging, and driver version mismatch resolution. Yet the message itself is calm, structured, and informative. It doesn't dwell on the struggle; it presents the results, documents the fix, and provides the comparison data that the user needs to make informed decisions.

For anyone operating large-scale GPU inference infrastructure, this message contains several lessons:

IOMMU mode matters. The difference between iommu=pt and full translation is not just a kernel parameter — it fundamentally changes how GPU DMA works. Always verify P2P behavior after changing IOMMU configuration.
Silent corruption is the hardest bug to find. The fact that can_device_access_peer returned True while transfers produced garbage data made this bug particularly treacherous. The assistant's systematic approach — testing actual data transfer rather than relying on capability queries — is the correct methodology.
Environment variables are infrastructure. The NCCL_P2P_DISABLE=1 fix, embedded in sitecustomize.py and the systemd service file, represents a permanent documentation of a hardware limitation. This is the kind of knowledge that must be preserved in configuration, not in README files.
Benchmark numbers need context. The comparison table in this message provides exactly the right context: not just raw throughput, but model quality, KV cache capacity, and the tradeoffs between quantization and precision. The message ends with a forward-looking note about MTP (Multi-Token Prediction) being "disabled for now since it requires extra_buffer scheduler + spec_v2." This acknowledges that the current deployment is a stable baseline, not the final state. The infrastructure is working, the model is serving, and the path forward is clear. That's exactly what a deployment complete message should communicate.