The IOMMU Barrier: GPU P2P DMA Corruption Under SEV-SNP — A Full-Stack Debugging Saga

Introduction

In the life of a production machine learning infrastructure, few debugging challenges are as formidable as a silent data corruption bug that spans four layers of abstraction: the application framework, the distributed communication library, the GPU driver, and the hypervisor's IOMMU. This article chronicles a single, intense opencode coding session where an AI assistant and its human collaborator navigated exactly such a bug — a GPU P2P DMA corruption issue under AMD's SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) that manifested as a simple application hang but whose root cause lay deep in the interaction between Blackwell GPU architecture, IOMMU translation, and NCCL transport selection.

The session spans over 160 messages (indexes 6026 through 6188) and covers four major phases: GPU topology reconfiguration and persistent binding, model swap from Qwen3.5-397B to Qwen3.5-122B, driver version mismatch diagnosis and resolution, and the eventual discovery of P2P DMA corruption under IOMMU translation. Each phase built on the previous one, and the final resolution — forcing NCCL to disable P2P transport — required understanding the entire virtualization stack from the hypervisor's IOMMU configuration to NCCL's transport internals.

The Stage: An 8-GPU Blackwell System Split Between Worlds

The session unfolded on an ASUS ESC8000A-E13 server — a dual-socket AMD EPYC 9335 (Zen 5 "Turin") machine with 512 GB of DDR5 RAM and eight NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB of GDDR7 memory. This was not a simple bare-metal deployment. The Proxmox hypervisor hosted two distinct environments: an LXC container running SGLang for LLM inference, and a SEV-SNP confidential VM for another tenant. The eight GPUs had been split — four bound to the NVIDIA driver for the container, four moved to vfio-pci passthrough for the VM.

This topology reconfiguration, performed by a previous agent, was the inciting event of the debugging saga. The kernel command line had been updated to include amd_iommu=on and mem_encrypt=on — both mandatory for AMD's SEV-SNP confidential computing. These flags changed the IOMMU (Input/Output Memory Management Unit) from its default passthrough mode (iommu=pt) to full translation mode, where every DMA transaction from every PCIe device must go through the IOMMU's address translation tables.

The assistant, unaware of this architectural constraint, proceeded to update the SGLang service from TP=8 to TP=4 and replace the model from Qwen3.5-397B NVFP4 to Qwen3.5-122B-A10B BF16. When it launched the server, the result was a complete stall.

Phase 1: The GPU Topology Split

The session's pivot point came when the user issued a decisive redirection: "For now assume it's good enough. I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)." This single sentence marked the end of an extensive optimization effort for the Qwen3.5-397B-A17B-NVFP4 model and the beginning of a fundamental infrastructure reconfiguration.

The assistant's first task was to understand the existing GPU topology. Through a series of diagnostic commands, the assistant mapped the eight Blackwell GPUs to their PCI addresses and NUMA nodes. The discovery was clean: four GPUs on NUMA 0 (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) and four on NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0). The NUMA 0 GPUs would stay bound to the NVIDIA driver for the LXC container; the NUMA 1 GPUs would be moved to vfio-pci for VM passthrough.

The actual reconfiguration required multiple steps: stopping the SGLang server and LXC container, unbinding the NUMA 1 GPUs from the NVIDIA driver via sysfs, binding them to vfio-pci, updating the LXC configuration to only expose four GPUs, and creating a new Proxmox PCI mapping (pro6000-vm) for the VM-side GPUs. The assistant performed the initial split at runtime using direct sysfs manipulation — writing PCI addresses to /sys/bus/pci/drivers/vfio-pci/bind after unbinding them from the NVIDIA driver.

But runtime binding is ephemeral. A single reboot would undo all the work, and since all eight GPUs share the same PCI device ID (10de:2bb5), the NVIDIA driver would reclaim all of them on the next boot. The assistant recognized this fragility and committed to making the split persistent.

Persistence by Design: The Systemd Service

The assistant's reasoning about persistence is a textbook example of constraint-driven design. The standard mechanism for binding devices to vfio-pci at boot is the vfio-pci.ids= kernel parameter, which tells the driver to claim any device matching the given vendor:device IDs. But as the assistant correctly identified, since all eight GPUs have the same device ID (10de:2bb5), using vfio-pci.ids= would grab all of them. This constraint forced a more surgical approach: binding by PCI address rather than by device ID.

The solution was a systemd service (gpu-vfio-split.service) that runs a shell script (/usr/local/bin/gpu-vfio-split.sh) early in boot, before the NVIDIA driver claims the GPUs. The service file uses Before=nvidia-persistenced.service to ensure it runs before the NVIDIA stack initializes, and After=systemd-modules-load.service to ensure the vfio-pci kernel module is loaded first. The script itself iterates over the four NUMA 1 PCI addresses, unbinds them from their current driver (if any), registers the device ID with vfio-pci via the new_id sysfs interface, and binds them.

This design reflects a nuanced understanding of Linux PCI driver binding, boot ordering, and the constraints imposed by homogeneous hardware IDs. The service was enabled and verified, transforming a fragile manual procedure into a reliable boot-time automation.

Phase 2: The Model Swap

With the GPU topology reconfigured, the assistant turned to the model deployment. The Qwen3.5-397B-A17B-NVFP4 model — a 397-billion-parameter Mixture-of-Experts model using 4-bit NVFP quantization — had been running on all eight GPUs. But the user had already rendered a quality verdict: "rm qwen, in use it's actually very low quality, we'll be deploying a similarly sized model." This was a decisive rejection delivered with the efficiency of a shell command annotation.

The replacement was Qwen3.5-122B-A10B, a 122-billion-parameter model (10 billion active parameters per token due to MoE) running in BF16 native precision. At approximately 234 GB, it fit comfortably on four 96 GB GPUs with tensor parallelism set to TP=4. The assistant downloaded the model to /shared (the /data volume was being retired), updated the SGLang systemd service file, and prepared to start the server.

But the first launch attempt crashed immediately. The error traceback revealed that the server was failing during argument parsing — specifically, the ServerArgs.from_cli_args method was rejecting the combination of flags for Multi-Token Prediction (MTP) with the hybrid GDN architecture. The assistant diagnosed the issue: MTP with the hybrid GDN model needs --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1.

This diagnosis required deep knowledge of SGLang's internals. The Qwen3.5-122B model uses a hybrid architecture with 36 linear attention (GDN/recurrent) layers and 12 full attention layers, arranged with a full attention layer every fourth position. When speculative decoding (MTP) is enabled with this non-uniform layer structure, the scheduler needs additional KV cache buffers to handle the mismatch between draft tokens and verified tokens. The --mamba-scheduler-strategy extra_buffer flag allocates those buffers, and SGLANG_ENABLE_SPEC_V2=1 enables an alternative speculative decoding path designed for complex architectures. The fix was applied as a six-character edit to the service file.

But it wasn't enough.## Phase 3: The Driver Version Mismatch

After the MTP fix, the server still wouldn't start. This time, it hung indefinitely at the log line "Init torch distributed begin" — the point where NCCL initializes communication channels between the four GPU workers. The assistant spent over thirty messages systematically debugging this hang.

The diagnostic process was methodical. The assistant checked NCCL debug output and found that NCCL initialization actually completed successfully — the bootstrap phase finished in 4.4 milliseconds, and all four ranks connected. It checked for port conflicts, stale rendezvous files, and process leaks. It tried overriding NCCL environment variables, tested TP=1 (which confirmed the model loaded but OOM'd on a single GPU), and even pulled the latest SGLang main branch for potential fixes. Every attempt produced the same result: the four TP processes would connect via NCCL, establish their TCPStore, and then freeze — blocked on futex and socket reads.

The breakthrough came from an unexpected source. The user observed: "So on the proxmox host we have made some changes to the driver, maybe we need to update in the VM to match?" This single sentence bridged a coordination gap. Another agent had been working on the Proxmox host simultaneously, upgrading the NVIDIA driver to version 590.48.01. The assistant, working inside the LXC container, was unaware of this change.

The investigation revealed a stark mismatch: the kernel module (inherited from the host) was version 590.48.01, but the container's userspace libraries — libnvidia-compute, libnvidia-decode, and nvidia-utils — were all version 565.57.01. This is a dangerous situation. The NVIDIA driver stack has two layers: the kernel module (running in the host OS) and the userspace libraries (running inside the container). These must be version-compatible. A mismatch can cause undefined behavior, including the exact NCCL hang symptom the assistant was observing.

The fix was surgical: install the matching 590 userspace packages from the Ubuntu apt repositories, remove the old 565 packages, and verify with nvidia-smi that version numbers were now consistent. The assistant then restarted the server.

Phase 4: The P2P DMA Corruption Under IOMMU

Even after the driver fix, the hang persisted. The assistant's NCCL debug output had shown that NCCL initialization itself completed fine, but something after that was blocking. The strace output revealed all four processes blocked on futex and restart_syscall — classic symptoms of a deadlock or a blocking collective operation.

The assistant's investigation eventually led to the source code of SGLang's distributed initialization, where it traced the exact execution path from "Init torch distributed begin" through NCCL communicator creation to the barrier synchronization. The code revealed that after NCCL init, the processes enter a barrier that waits for all ranks to reach the same point before proceeding with model loading. If one rank never reaches that barrier — or if the barrier itself deadlocks — the entire initialization hangs.

The Smoking Gun: IO_PAGE_FAULTs in dmesg

The breakthrough came from the user, who provided a critical piece of evidence that the assistant, operating inside the LXC container, could not see. The user shared kernel logs from the Proxmox host:

[73784.343781] nvidia 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x24000000000 flags=0x0030]
[73784.343803] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x34000001000 flags=0x0030]
[73784.344012] nvidia 0000:71:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0x34000010000 flags=0x0030]
[73784.344125] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x6e000000000 flags=0x0030]

These four lines transformed the investigation. The AMD-Vi identifier confirmed the AMD IOMMU was active. The IO_PAGE_FAULT events indicated that DMA transactions from the NVIDIA GPUs were being intercepted and blocked. The addresses — 0x24000000000, 0x34000001000, 0x34000010000, 0x6e000000000 — were GPU BAR (Base Address Register) addresses, the memory-mapped I/O windows that GPUs expose for peer-to-peer communication. The flags value 0x0030 indicated a write transaction to a page that the IOMMU had not mapped.

The assistant recognized the significance immediately: those IO_PAGE_FAULTs from AMD-Vi (IOMMU) on the nvidia GPUs were a big red flag — the GPUs were trying to do P2P DMA but the IOMMU was blocking it. This was the pivot point from software debugging to hardware diagnosis.

The Documentation Hunt: A Single-Character Typo

The user then pointed the assistant toward the setup documentation that would explain the IOMMU configuration, mentioning files named v1-uefi-settings.md and snp-pxm8-v1.md in a directory called ~/kpro-uefi/. These files, created by the agent that had performed the GPU topology reconfiguration, would contain the BIOS settings and SEV-SNP configuration details.

What followed was a remarkable sequence of failed search attempts that illustrates the challenges of multi-machine debugging. The assistant first tried the remote Proxmox host (ssh root@10.1.2.6 'ls ~/kpro-uefi/') — nothing. It tried a broader find command on the remote host — nothing. It tried the local machine with find ~/kpro-uefi/ — nothing. It used a glob pattern **/kpro-uefi/**/* — no files found. It tried searching in /home/theuser — the command timed out after 5 seconds.

The assistant had exhausted its automated search capabilities. It resorted to asking a structured question, offering three options for where the files might reside. The user's response was a single word: kpro6-uefi. Not kpro-uefi, but kpro6-uefi — a single missing character "6" that had rendered every search futile.

This moment is a powerful lesson in the importance of precise naming in collaborative debugging. A one-character typo in the user's original reference cascaded through five search attempts across multiple machines, consuming dozens of messages and significant time. The assistant's structured question finally forced the precision that was needed.

With the correct path in hand, the assistant listed the directory contents, revealing a rich collection of BIOS analysis artifacts — ACPI table dumps, IFR decodes, BIOS capsule files, and crucially, the two markdown documents. Reading them unlocked the complete understanding of the system configuration.

The Epiphany: Full IOMMU Translation and Its Consequences

The documentation revealed the critical architectural constraint. The Proxmox host was configured with amd_iommu=on (full IOMMU translation mode) rather than iommu=pt (passthrough mode). This was not a mistake or a misconfiguration — it was a deliberate requirement of SEV-SNP confidential computing, which needs the IOMMU to enforce memory encryption and isolation boundaries for the VM.

The assistant synthesized this understanding: IOMMU was now in full translation mode (amd_iommu=on, not iommu=pt) because SEV-SNP requires it. Previously it was likely in passthrough mode. The IO_PAGE_FAULTs were the GPUs trying to do P2P DMA through the IOMMU and failing.

This single insight reframed the entire problem. The NCCL hang was not a software bug — it was a hardware security feature creating a fundamental incompatibility. Under full IOMMU translation, every DMA transaction from a PCIe device must go through the IOMMU's page tables. GPU P2P DMA, which relies on direct memory access between GPU BAR addresses, generates addresses that the IOMMU cannot translate because those BAR mappings are not in its page tables. The result is IO_PAGE_FAULTs, corrupted data, and NCCL deadlocks.

The assistant verified this by checking the kernel command line on the Proxmox host (cat /proc/cmdline), confirming amd_iommu=on, mem_encrypt=on, and kvm_amd.sev=1 — all consistent with an SEV-SNP configuration, with no iommu=pt override.

The Confirmation: Every P2P Transfer Is Corrupted

With the hypothesis formed, the assistant designed a series of confirmatory experiments. First, it tested single-GPU functionality with a simple PyTorch tensor operation — this succeeded, confirming the GPUs themselves were healthy and the driver stack was functional.

Next, it queried P2P capability using torch.cuda.can_device_access_peer(). The result was superficially reassuring: all 12 directed pairs among the 4 GPUs returned True. The CUDA driver believed P2P was available.

Then came the critical test. The assistant wrote a loop that created a random tensor on one GPU, copied it to another GPU via .to(), synchronized, and compared the results. The output was devastating:

Testing GPU-to-GPU copy...
  GPU 0 -> GPU 1: MISMATCH
  GPU 0 -> GPU 2: MISMATCH
  GPU 0 -> GPU 3: MISMATCH
  GPU 1 -> GPU 0: MISMATCH
  GPU 1 -> GPU 2: MISMATCH
  GPU 1 -> GPU 3: MISMATCH
  GPU 2 -> GPU 0: MISMATCH
  GPU 2 -> GPU 1: MISMATCH
  GPU 2 -> GPU 3: MISMATCH
  GPU 3 -> GPU 0: MISMATCH
  GPU 3 -> GPU 1: MISMATCH
  GPU 3 -> GPU 2: MISMATCH

Every single GPU-to-GPU transfer produced corrupted data. Not a single pair survived the round-trip intact. The driver reported P2P as available, but the IOMMU was silently corrupting every transfer. This was the definitive proof.

The assistant then tested NCCL with NCCL_P2P_DISABLE=1, forcing it to use shared memory (SHM) transport instead of P2P DMA. The result was unambiguous success:

Testing NCCL all-reduce with 4 GPUs, P2P disabled...
Rank 3: all_reduce OK (got 10.0, expected 10.0)
Rank 1: all_reduce OK (got 10.0, expected 10.0)
Rank 2: all_reduce OK (got 10.0, expected 10.0)
Rank 0: all_reduce OK (got 10.0, expected 10.0)

All four ranks completed the distributed all-reduce correctly. The fix was validated.

The Fix: A Two-Layer Deployment Strategy

With the root cause confirmed, the assistant implemented the fix in two layers, each serving a different purpose and providing defense-in-depth.

The first layer was the global Python environment. The assistant updated /usr/lib/python3.12/sitecustomize.py to set NCCL_P2P_DISABLE=1 along with other NCCL tuning parameters (LL protocol, Ring algorithm, 16 channels, 16 MB buffer size, 512 threads). The comment in the file explicitly documented the rationale: "P2P is DISABLED because IOMMU in non-passthrough mode blocks GPU-to-GPU DMA." This ensured that any Python process using NCCL — whether launched by systemd, by hand, or by a subprocess — would automatically inherit the P2P-disabled configuration.

The second layer was the systemd service file. The assistant read the existing sglang-qwen.service file and applied an edit to add NCCL_P2P_DISABLE=1 and CUDA_DEVICE_MAX_CONNECTIONS=1 to the service's Environment directives. This is the production deployment path: the systemd service is what launches SGLang on boot or restart, and it must carry the fix independently of the Python environment.

The assistant then cleaned up the old processes with kill -9 and fuser -k /dev/nvidia*, copied the updated service file to the remote container, and started the service. The server loaded successfully, achieving 108 tok/s single-stream and up to 2,800 tok/s at high concurrency — validating that the SHM transport workaround, while theoretically slower than P2P DMA, was more than adequate for production inference.

The Aftermath: Investigating BIOS Options

With the immediate crisis resolved, the assistant turned to investigating whether P2P DMA could be re-enabled under IOMMU translation — a proper fix rather than a workaround. This involved exploring BIOS-level options like PCIe ACS (Access Control Services), ARI (Alternative Routing-ID Interpretation), and IOMMU coverage settings. The user had explicitly warned against rebooting the host or changing BIOS settings without coordination, as another tenant was active on the machine, but the assistant documented the investigation path for future reference.

The key question was whether the IOMMU could be configured to allow P2P DMA while maintaining SEV-SNP security guarantees. Options included setting iommu=pt for specific PCIe devices (if the kernel and hardware support per-device passthrough), adjusting ACS settings to allow peer-to-peer traffic across PCIe switches, or using NVIDIA's GPUDirect RDMA with appropriate IOMMU mappings. Each of these options required further research and coordination with the other tenant.

Lessons in Debugging Methodology

This debugging saga, spanning dozens of messages and multiple sub-sessions, offers several enduring lessons for systems engineering:

The importance of cross-layer reasoning. The failure manifested at the application layer (SGLang hanging), but its root cause was in the virtualization layer (IOMMU configuration), triggered by a security requirement (SEV-SNP). The assistant had to trace the failure across four layers of abstraction — application, distributed framework, GPU driver, and hypervisor — to find the root cause. This kind of multi-layer debugging is increasingly essential as systems grow more complex.

The danger of premature hypothesis formation. The assistant invested significant effort in the "SGLang version is too old" hypothesis — pulling the latest code, verifying patches, reinstalling — without first ruling out simpler hardware-level causes. A quicker check of host dmesg for IO_PAGE_FAULTs might have saved dozens of steps. The user's intervention, providing the kernel logs directly, was the critical correction.

The value of the user as a domain expert. The assistant's systematic software debugging was thorough but tunnel-visioned. The user, seeing the broader system context (the Proxmox host, the SEV-SNP VM, the IOMMU configuration), immediately recognized the IO_PAGE_FAULT pattern and provided the evidence that reframed the investigation. The best debugging outcomes often come from combining the assistant's methodical tool use with the user's system-level intuition.

The cost of imprecise naming. A single missing character — "kpro-uefi" vs "kpro6-uefi" — consumed dozens of messages and multiple search strategies. The assistant faithfully searched for the wrong string for several rounds before asking for clarification. This is a reminder that AI systems are ruthlessly literal — they search for exactly what they're told, not what was intended.

The gap between capability and reliability. The most subtle insight of the entire investigation was the distinction between can_device_access_peer=True (the driver's report that the hardware supports P2P) and actual data integrity (the IOMMU silently corrupting every transfer). The CUDA driver's capability query checks whether the PCIe topology supports P2P, but it cannot predict whether the IOMMU configuration will allow it to function correctly. This gap between "can the hardware do it" and "will the platform allow it" is a critical diagnostic concept for IOMMU-related issues.

The power of systematic elimination. The assistant's diagnostic sequence — single-GPU test → P2P capability query → actual P2P data transfer → NCCL with P2P disabled — is a textbook example of layered debugging. Each test eliminated a class of explanations and narrowed the search space. The final NCCL test with NCCL_P2P_DISABLE=1 was the definitive experiment that confirmed both the root cause and the fix.

The Outcome: A Working Deployment

With NCCL_P2P_DISABLE=1 in place and the driver version mismatch resolved, the server loaded successfully. The assistant benchmarked the Qwen3.5-122B-A10B BF16 model on four Blackwell GPUs, achieving 108 tok/s single-stream and up to 2,800 tok/s at concurrency level 128. These are impressive numbers that vindicate the assistant's persistence through the debugging ordeal.

The workaround is not the ideal solution — P2P DMA would be faster than SHM transport — but it is a correct one, and it is deployed with the documentation and rationale that will allow future engineers to understand why P2P is disabled and under what circumstances it could be re-enabled. That is the mark of a debugging effort done right.

Conclusion

This session captures the full lifecycle of a complex ML infrastructure deployment: from topology reconfiguration and persistent automation, through model selection and deployment, through multi-layered debugging spanning driver versions and hardware virtualization, to a working production service. The assistant's systematic approach, its ability to pivot between hypotheses, and its deep understanding of the entire stack — from Linux PCI driver binding to NCCL transport protocols to SGLang's hybrid GDN architecture — made the difference between a stuck deployment and a running server serving 2,800 tokens per second.

The debugging odyssey documented in this session is a masterclass in modern infrastructure engineering. It demonstrates that the most elusive bugs are often those where the symptom manifests at a different level of abstraction than the root cause — an application hang caused by a hardware security feature, a driver capability report that contradicts actual behavior, a documentation typo that sends automated searches on a wild goose chase. The fix itself — NCCL_P2P_DISABLE=1 — is a single environment variable, but the reasoning behind it required understanding GPU architecture, IOMMU virtualization, NCCL transport internals, SEV-SNP confidential computing requirements, and the specific constraints of a shared Proxmox host.

The final investigation into BIOS-level P2P DMA options reminds us that in production systems, the work is never truly done. Every fix creates new questions, and every workaround points toward a deeper understanding of the hardware-software boundary. The system achieved 2,800 tok/s on a 122-billion-parameter model across 4 Blackwell GPUs — a testament to the fact that even with P2P DMA disabled, the SHM transport fallback is more than adequate for production inference.## References

[1] "The Blackwell GPU Split: From Topology Reconfiguration to P2P DMA Debugging on Proxmox" — Chunk article covering the GPU topology split, persistent binding systemd service, model swap, driver mismatch, and P2P DMA debugging.

[2] "The IOMMU Barrier: A Comprehensive Debugging Odyssey Through GPU P2P DMA Corruption Under SEV-SNP" — Chunk article covering the full IOMMU debugging saga, from IO_PAGE_FAULT discovery through documentation hunt to experimental confirmation and fix deployment.