The IOMMU Barrier: A Comprehensive Debugging Odyssey Through GPU P2P DMA Corruption Under SEV-SNP

Introduction

In the span of a single opencode coding session, an AI assistant and its human collaborator navigated one of the most challenging debugging scenarios in modern infrastructure engineering: a silent data corruption bug that spanned GPU architecture, IOMMU virtualization, NCCL transport selection, and distributed inference initialization. The problem manifested as a simple hang — SGLang would freeze during init_torch_distributed — but its root cause lay deep in the interaction between AMD's SEV-SNP confidential computing technology and NVIDIA's GPU-to-GPU peer-to-peer DMA. This article synthesizes the full arc of that investigation, from topology reconfiguration through root cause diagnosis, documentation archaeology, experimental confirmation, and ultimately, the deployment of a pragmatic workaround that restored the system to full production capability.

The Stage: An 8-GPU Blackwell System Split Between Worlds

The session unfolded on an ASUS ESC8000A-E13 server — a dual-socket AMD EPYC 9335 (Zen 5 "Turin") machine with 512 GB of DDR5 RAM and eight NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB of GDDR7 memory. This was not a simple bare-metal deployment. The Proxmox hypervisor hosted two distinct environments: an LXC container running SGLang for LLM inference, and a SEV-SNP (Secure Encrypted Virtualization — Secure Nested Paging) confidential VM for another tenant. The eight GPUs had been split — four bound to the NVIDIA driver for the container, four moved to vfio-pci passthrough for the VM.

This topology reconfiguration, performed by a previous agent, was the inciting event of the debugging saga. The kernel command line had been updated to include amd_iommu=on and mem_encrypt=on — both mandatory for AMD's SEV-SNP confidential computing. These flags changed the IOMMU (Input/Output Memory Management Unit) from its default passthrough mode (iommu=pt) to full translation mode, where every DMA transaction from every PCIe device must go through the IOMMU's address translation tables.

The assistant, unaware of this architectural constraint, proceeded to update the SGLang service from TP=8 to TP=4 and replace the model from Qwen3.5-397B NVFP4 to Qwen3.5-122B-A10B BF16. When it launched the server, the result was a complete stall.

The Hang: A Distributed Initialization Deadlock

The symptom was maddeningly consistent. Every attempt to start SGLang with --tp 4 produced the same log output:

[2026-03-09 12:32:57 TP2] Mamba selective_state_update backend initialized: triton
[2026-03-09 12:32:57 TP2] Init torch distributed begin.

And then — silence. The four TP worker processes would establish NCCL sockets, connect to the TCPStore, and then freeze indefinitely on futex and restart_syscall system calls. GPU memory sat flat at approximately 1,100 MiB per card — the CUDA context baseline, far below the 234 GB the model required.

The assistant's initial response was a thorough but ultimately misdirected software-level investigation. It checked NCCL environment variables, verified port availability, ran NCCL debug traces, killed stale processes, tried TP=1 to confirm the model itself loaded (it did, until OOM on a single GPU), overrode NCCL settings from sitecustomize.py, attached strace to the hung processes and found them all blocked on futex syscalls, inspected socket state and confirmed NCCL connections were established, read the SGLang source code to understand what "Init torch distributed begin" actually does, verified the TCPStore port was listening and all four ranks were connected, and finally concluded the hang must be in initialize_model_parallel rather than NCCL init. It even pulled the latest SGLang main branch and reinstalled from source, suspecting a version compatibility issue with the newly released Qwen3.5 model.

Each of these steps was reasonable. Each eliminated a hypothesis. But none could fix the problem because none addressed the true root cause — a root cause that lived not in the software stack, but in the hardware virtualization layer.

The Smoking Gun: IO_PAGE_FAULTs in dmesg

The breakthrough came from the user, who provided a critical piece of evidence that the assistant, operating inside the LXC container, could not see. In message <msg id=6191>, the user shared kernel logs from the Proxmox host:

[73784.343781] nvidia 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x24000000000 flags=0x0030]
[73784.343803] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x34000001000 flags=0x0030]
[73784.344012] nvidia 0000:71:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0x34000010000 flags=0x0030]
[73784.344125] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x6e000000000 flags=0x0030]

These four lines transformed the investigation. The AMD-Vi identifier confirmed the AMD IOMMU was active. The IO_PAGE_FAULT events indicated that DMA transactions from the NVIDIA GPUs were being intercepted and blocked. The addresses — 0x24000000000, 0x34000001000, 0x34000010000, 0x6e000000000 — were GPU BAR (Base Address Register) addresses, the memory-mapped I/O windows that GPUs expose for peer-to-peer communication. The flags value 0x0030 indicated a write transaction to a page that the IOMMU had not mapped.

The assistant recognized the significance immediately in <msg id=6193>: "Those IO_PAGE_FAULTs from AMD-Vi (IOMMU) on the nvidia GPUs are a big red flag — the GPUs are trying to do P2P DMA but IOMMU is blocking it." This was the pivot point from software debugging to hardware diagnosis.

The Documentation Hunt: A Single-Character Typo That Cost Dozens of Messages

The user then pointed the assistant toward the setup documentation that would explain the IOMMU configuration. In <msg id=6196>, the user wrote: "~/kpro-uefi/ esp. v1-uefi-settings.md and snp-pxm8-v1.md contain notes from the agent that did this setup." These files, created by the agent that had performed the GPU topology reconfiguration, would contain the BIOS settings and SEV-SNP configuration details.

What followed was a remarkable sequence of failed search attempts that illustrates the challenges of multi-machine debugging. The assistant first tried the remote Proxmox host (ssh root@10.1.2.6 'ls ~/kpro-uefi/') — nothing. It tried a broader find command on the remote host — nothing. It tried the local machine with find ~/kpro-uefi/ — nothing. It used a glob pattern **/kpro-uefi/**/* — no files found. It tried searching in /home/theuser — the command timed out after 5 seconds.

The assistant had exhausted its automated search capabilities. It resorted to asking a structured question via the question tool, offering three options for where the files might reside. The user's response in <msg id=6206> was a single word: kpro6-uefi. Not kpro-uefi, but kpro6-uefi — a single missing character "6" that had rendered every search futile.

This moment, captured in articles [18] and [19], is a powerful lesson in the importance of precise naming in collaborative debugging. A one-character typo in the user's original reference cascaded through five search attempts across multiple machines, consuming dozens of messages and significant time. The assistant's structured question finally forced the precision that was needed.

With the correct path in hand, the assistant listed the directory contents in <msg id=6207>, revealing a rich collection of BIOS analysis artifacts — ACPI table dumps, IFR decodes, BIOS capsule files, and crucially, the two markdown documents. Reading them in <msg id=6208> unlocked the complete understanding of the system configuration.

The Epiphany: Full IOMMU Translation and Its Consequences

The documentation revealed the critical architectural constraint. The Proxmox host was configured with amd_iommu=on (full IOMMU translation mode) rather than iommu=pt (passthrough mode). This was not a mistake or a misconfiguration — it was a deliberate requirement of SEV-SNP confidential computing, which needs the IOMMU to enforce memory encryption and isolation boundaries for the VM.

The assistant synthesized this understanding in <msg id=6209>: "IOMMU is now in full translation mode (amd_iommu=on, not iommu=pt) because SEV-SNP requires it. Previously it was likely in passthrough mode. The IO_PAGE_FAULTs are the GPUs trying to do P2P DMA through the IOMMU and failing."

This single insight reframed the entire problem. The NCCL hang was not a software bug — it was a hardware security feature creating a fundamental incompatibility. Under full IOMMU translation, every DMA transaction from a PCIe device must go through the IOMMU's page tables. GPU P2P DMA, which relies on direct memory access between GPU BAR addresses, generates addresses that the IOMMU cannot translate because those BAR mappings are not in its page tables. The result is IO_PAGE_FAULTs, corrupted data, and NCCL deadlocks.

The assistant verified this by checking the kernel command line on the Proxmox host (cat /proc/cmdline), confirming amd_iommu=on, mem_encrypt=on, and kvm_amd.sev=1 — all consistent with an SEV-SNP configuration, with no iommu=pt override.

The Confirmation: Every P2P Transfer Is Corrupted

With the hypothesis formed, the assistant designed a series of confirmatory experiments. First, it tested single-GPU functionality with a simple PyTorch tensor operation — this succeeded, confirming the GPUs themselves were healthy and the driver stack was functional.

Next, it queried P2P capability using torch.cuda.can_device_access_peer(). The result was superficially reassuring: all 12 directed pairs among the 4 GPUs returned True. The CUDA driver believed P2P was available.

Then came the critical test. The assistant wrote a loop that created a random tensor on one GPU, copied it to another GPU via .to(), synchronized, and compared the results. The output, documented in <msg id=6216>, was devastating:

Testing GPU-to-GPU copy...
  GPU 0 -> GPU 1: MISMATCH
  GPU 0 -> GPU 2: MISMATCH
  GPU 0 -> GPU 3: MISMATCH
  GPU 1 -> GPU 0: MISMATCH
  GPU 1 -> GPU 2: MISMATCH
  GPU 1 -> GPU 3: MISMATCH
  GPU 2 -> GPU 0: MISMATCH
  GPU 2 -> GPU 1: MISMATCH
  GPU 2 -> GPU 3: MISMATCH
  GPU 3 -> GPU 0: MISMATCH
  GPU 3 -> GPU 1: MISMATCH
  GPU 3 -> GPU 2: MISMATCH

Every single GPU-to-GPU transfer produced corrupted data. Not a single pair survived the round-trip intact. The driver reported P2P as available, but the IOMMU was silently corrupting every transfer. This was the definitive proof.

The assistant then tested NCCL with NCCL_P2P_DISABLE=1, forcing it to use shared memory (SHM) transport instead of P2P DMA. The result, in <msg id=6219>, was unambiguous success:

Testing NCCL all-reduce with 4 GPUs, P2P disabled...
Rank 3: all_reduce OK (got 10.0, expected 10.0)
Rank 1: all_reduce OK (got 10.0, expected 10.0)
Rank 2: all_reduce OK (got 10.0, expected 10.0)
Rank 0: all_reduce OK (got 10.0, expected 10.0)

All four ranks completed the distributed all-reduce correctly. The fix was validated.

The Fix: A Two-Layer Deployment Strategy

With the root cause confirmed, the assistant implemented the fix in two layers, each serving a different purpose and providing defense-in-depth.

The first layer was the global Python environment. The assistant updated /usr/lib/python3.12/sitecustomize.py to set NCCL_P2P_DISABLE=1 along with other NCCL tuning parameters (LL protocol, Ring algorithm, 16 channels, 16 MB buffer size, 512 threads). The comment in the file explicitly documented the rationale: "P2P is DISABLED because IOMMU in non-passthrough mode blocks GPU-to-GPU DMA." This ensured that any Python process using NCCL — whether launched by systemd, by hand, or by a subprocess — would automatically inherit the P2P-disabled configuration.

The second layer was the systemd service file. The assistant read the existing sglang-qwen.service file and applied an edit to add NCCL_P2P_DISABLE=1 and CUDA_DEVICE_MAX_CONNECTIONS=1 to the service's Environment directives. This is the production deployment path: the systemd service is what launches SGLang on boot or restart, and it must carry the fix independently of the Python environment.

The assistant then cleaned up the old processes with kill -9 and fuser -k /dev/nvidia*, copied the updated service file to the remote container, and started the service. The server loaded successfully, achieving 108 tok/s single-stream and up to 2,800 tok/s at high concurrency — validating that the SHM transport workaround, while theoretically slower than P2P DMA, was more than adequate for production inference.

The Aftermath: Investigating BIOS Options

With the immediate crisis resolved, the assistant turned to investigating whether P2P DMA could be re-enabled under IOMMU translation — a proper fix rather than a workaround. This involved exploring BIOS-level options like PCIe ACS (Access Control Services), ARI (Alternative Routing-ID Interpretation), and IOMMU coverage settings. The user had explicitly warned against rebooting the host or changing BIOS settings without coordination, as another tenant was active on the machine, but the assistant documented the investigation path for future reference.

The key question was whether the IOMMU could be configured to allow P2P DMA while maintaining SEV-SNP security guarantees. Options included setting iommu=pt for specific PCIe devices (if the kernel and hardware support per-device passthrough), adjusting ACS settings to allow peer-to-peer traffic across PCIe switches, or using NVIDIA's GPUDirect RDMA with appropriate IOMMU mappings. Each of these options required further research and coordination with the other tenant.

Lessons in Debugging Methodology

This debugging saga, spanning dozens of messages and multiple sub-sessions, offers several enduring lessons for systems engineering:

The importance of cross-layer reasoning. The failure manifested at the application layer (SGLang hanging), but its root cause was in the virtualization layer (IOMMU configuration), triggered by a security requirement (SEV-SNP). The assistant had to trace the failure across four layers of abstraction — application, distributed framework, GPU driver, and hypervisor — to find the root cause. This kind of multi-layer debugging is increasingly essential as systems grow more complex.

The danger of premature hypothesis formation. The assistant invested significant effort in the "SGLang version is too old" hypothesis — pulling the latest code, verifying patches, reinstalling — without first ruling out simpler hardware-level causes. A quicker check of host dmesg for IO_PAGE_FAULTs might have saved dozens of steps. The user's intervention, providing the kernel logs directly, was the critical correction.

The value of the user as a domain expert. The assistant's systematic software debugging was thorough but tunnel-visioned. The user, seeing the broader system context (the Proxmox host, the SEV-SNP VM, the IOMMU configuration), immediately recognized the IO_PAGE_FAULT pattern and provided the evidence that reframed the investigation. The best debugging outcomes often come from combining the assistant's methodical tool use with the user's system-level intuition.

The cost of imprecise naming. A single missing character — "kpro-uefi" vs "kpro6-uefi" — consumed dozens of messages and multiple search strategies. The assistant faithfully searched for the wrong string for several rounds before asking for clarification. This is a reminder that AI systems are ruthlessly literal — they search for exactly what they're told, not what was intended.

The gap between capability and reliability. The most subtle insight of the entire investigation was the distinction between can_device_access_peer=True (the driver's report that the hardware supports P2P) and actual data integrity (the IOMMU silently corrupting every transfer). The CUDA driver's capability query checks whether the PCIe topology supports P2P, but it cannot predict whether the IOMMU configuration will allow it to function correctly. This gap between "can the hardware do it" and "will the platform allow it" is a critical diagnostic concept for IOMMU-related issues.

The power of systematic elimination. The assistant's diagnostic sequence — single-GPU test → P2P capability query → actual P2P data transfer → NCCL with P2P disabled — is a textbook example of layered debugging. Each test eliminated a class of explanations and narrowed the search space. The final NCCL test with NCCL_P2P_DISABLE=1 was the definitive experiment that confirmed both the root cause and the fix.

Conclusion

The debugging odyssey documented in this chunk of the opencode session is a masterclass in modern infrastructure engineering. It demonstrates that the most elusive bugs are often those where the symptom manifests at a different level of abstraction than the root cause — an application hang caused by a hardware security feature, a driver capability report that contradicts actual behavior, a documentation typo that sends automated searches on a wild goose chase.

The fix itself — NCCL_P2P_DISABLE=1 — is a single environment variable, but the reasoning behind it required understanding GPU architecture, IOMMU virtualization, NCCL transport internals, SEV-SNP confidential computing requirements, and the specific constraints of a shared Proxmox host. The assistant's methodical approach — hypothesize, test, isolate, fix, document — turned a silent data corruption bug into a clean, documented workaround that restored the distributed inference pipeline to full operation.

In the end, the system achieved 2,800 tok/s on a 122-billion-parameter model across 4 Blackwell GPUs — a testament to the fact that even with P2P DMA disabled, the SHM transport fallback is more than adequate for production inference. The workaround is not the ideal solution, but it is a correct one, and it is deployed with the documentation and rationale that will allow future engineers to understand why P2P is disabled and under what circumstances it could be re-enabled. That is the mark of a debugging effort done right.