The IO_PAGE_FAULT Breakthrough: Diagnosing GPU P2P DMA Corruption Under IOMMU

Message Overview

The subject message, <msg id=6193>, is a brief but pivotal response from the assistant in an extended debugging session. It contains exactly one paragraph of analysis followed by a single bash command:

Those IO_PAGE_FAULTs from AMD-Vi (IOMMU) on the nvidia GPUs are a big red flag — the GPUs are trying to do P2P DMA but IOMMU is blocking it. This likely happened when the other agent changed IOMMU/vfio settings for the VM passthrough. Let me run some diagnostics.

>

[bash] ssh root@10.1.230.174 'kill -9 $(pgrep -f sglang) 2>/dev/null; fuser -k /dev/nvidia* 2>/dev/null; echo "cleaned"'

On its surface, this looks like a simple acknowledgment and cleanup command. But in the context of the preceding 30+ messages of frustrating, dead-end debugging, this message represents the moment everything clicked — the diagnostic breakthrough that transformed a mysterious hang into a well-understood hardware compatibility issue.

The Debugging Hell That Preceded This Message

To understand why this message matters, one must appreciate the depth of confusion that came before it. The assistant had been trying to deploy the Qwen3.5-122B-A10B BF16 model (a 234 GB MoE model) across 4 RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism. The server consistently hung at the same point: after printing "Init torch distributed begin," it would stall indefinitely, never completing initialization.

What followed was a classic debugging spiral. The assistant checked NCCL ports, verified socket connectivity, ran strace to see what system calls the processes were blocking on, tried different NCCL environment variables (NCCL_PROTO, NCCL_ALGO, NCCL_P2P_LEVEL), tested with TP=1 to confirm the model itself loaded, pulled the latest SGLang source and rebuilt it, and even verified that the SM120 Blackwell patches were still in place. Each attempt produced the same result: the four TP worker processes would connect via TCP, establish NCCL sockets, and then freeze indefinitely on futex and restart_syscall calls.

The strace output from <msg id=6172> showed all four TP workers blocked on futex(FUTEX_WAIT_BITSET) and restart_syscall — the classic signature of a deadlock or a barrier that never completes. The assistant had correctly identified that init_process_group was hanging, but the root cause remained elusive because all the symptoms pointed toward a software configuration issue: NCCL tuning parameters, port conflicts, or SGLang version incompatibility.

The User Provides the Missing Piece

The critical clue came from the user in <msg id=6191>, who provided host-side kernel logs showing IO_PAGE_FAULT events from the AMD-Vi (IOMMU) subsystem targeting the NVIDIA GPUs:

[73784.343781] nvidia 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x24000000000 flags=0x0030]
[73784.343803] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x34000001000 flags=0x0030]

These messages indicate that the IOMMU (Input-Output Memory Management Unit) on the AMD host was intercepting and blocking DMA (Direct Memory Access) operations from the GPUs. Each fault represents the GPU attempting to read or write to a memory address that the IOMMU translation tables do not permit. The flags=0x0030 pattern suggests these are write operations to unmapped addresses — exactly what happens when a GPU tries to perform P2P (peer-to-peer) DMA directly to another GPU's memory across the PCIe bus, and the IOMMU has not been configured to allow that specific transfer.

The Reasoning: Why This Message Was Written

The assistant's response in <msg id=6193> demonstrates a rapid and accurate diagnostic inference. The reasoning process visible in the message can be broken down into three steps:

Step 1: Pattern recognition. The assistant immediately identifies the IO_PAGE_FAULTs as "a big red flag." This is not hyperbole — in the context of NVIDIA GPUs on AMD platforms with IOMMU enabled, these faults are the canonical symptom of broken P2P DMA. The assistant has clearly encountered this pattern before, or has enough systems-level knowledge to recognize that IOMMU page faults on GPU DMA operations are never benign.

Step 2: Causal linking. The assistant connects the IOMMU faults to the NCCL hang that had consumed the previous 30+ messages. The reasoning is: NCCL uses P2P DMA for GPU-to-GPU communication within tensor parallelism. If IOMMU blocks every P2P transfer, the data never arrives, and NCCL's synchronization barriers never complete. This explains why init_torch_distributed hung — it was waiting for NCCL collective operations that were silently failing due to corrupted or dropped P2P data.

Step 3: Root cause attribution. The assistant correctly identifies the proximate cause: "This likely happened when the other agent changed IOMMU/vfio settings for the VM passthrough." This references the earlier reconfiguration in this segment where the Proxmox host's GPU topology was split, with 4 GPUs assigned to an SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) VM via vfio-pci passthrough. The SEV-SNP configuration requires full IOMMU translation (amd_iommu=on), which in turn breaks GPU P2P DMA because the IOMMU translation tables are not set up to allow peer-to-peer transfers between devices.

Assumptions Made in This Message

The assistant makes several assumptions, all of which turn out to be correct:

  1. The IO_PAGE_FAULTs are causal, not incidental. The assistant assumes these faults are the root cause of the NCCL hang, not a separate issue or a benign warning. This is a strong assumption — IOMMU page faults can sometimes be recoverable or non-fatal depending on the driver and hardware configuration. In this case, the assumption is validated by subsequent testing.
  2. P2P DMA is the specific mechanism being blocked. The assistant assumes the GPUs are trying to use P2P DMA (direct GPU-to-GPU memory access over PCIe) and that IOMMU is intercepting these transfers. This is a reasonable inference given that NCCL's default transport for same-node multi-GPU communication is P2P via NVLink or PCIe, and the Blackwell RTX PRO 6000 GPUs in this system are connected via PCIe (not NVLink).
  3. The VM passthrough configuration change is the trigger. The assistant assumes that the IOMMU behavior changed specifically because of the SEV-SNP VM setup done earlier in the segment. This is correct — enabling SEV-SNP requires amd_iommu=on, which activates full IOMMU translation for all PCIe devices, not just those passed through to the VM.
  4. Killing the stuck processes and cleaning GPU state is the right first step. The assistant assumes that the hung SGLang processes have left the GPUs in an inconsistent state and need to be fully cleaned before diagnostic tests can run.

No Mistakes — But an Important Limitation

The message itself contains no factual errors or incorrect assumptions. However, there is a notable limitation in the reasoning: the assistant correctly identifies that IOMMU is blocking P2P DMA, but it does not yet know why the IOMMU configuration is incompatible with P2P. The specific mechanism — whether it's the SEV-SNP IOMMU translation tables, the ACS (Access Control Services) PCIe capability, or the ARI (Alternative Routing-ID Interpretation) setting — is not yet determined. This is appropriate for a first-response diagnostic message; the assistant is signaling that it has identified the class of problem and is about to run targeted tests to narrow it down.

The message also does not yet propose a solution. The assistant's plan is to "run some diagnostics" first, which is the correct approach. The eventual fix — setting NCCL_P2P_DISABLE=1 to force NCCL to use shared memory (SHM) transport instead of P2P DMA — comes in the subsequent messages.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs knowledge in several domains:

IOMMU and AMD-Vi. The AMD-Vi (Virtualization Interface) is AMD's IOMMU implementation. It provides DMA remapping and interrupt remapping for PCIe devices, enabling secure device passthrough to virtual machines. When amd_iommu=on is set, the IOMMU intercepts all DMA transactions and translates device addresses to system addresses using I/O page tables. If a device attempts to DMA to an address not in its translation table, an IO_PAGE_FAULT is generated.

GPU P2P DMA. Peer-to-peer DMA allows one GPU to directly access another GPU's memory across the PCIe fabric without involving the CPU or system memory. This is critical for NCCL collectives in tensor-parallel inference, where each GPU needs to exchange intermediate activations with its peers. P2P DMA requires the PCIe switch to support peer-to-peer routing and the IOMMU to permit the transaction.

SEV-SNP and vfio-pci. SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) is AMD's confidential computing technology. When combined with vfio-pci device passthrough, it requires the IOMMU to enforce DMA isolation — each device can only DMA to memory explicitly mapped for it. This security mechanism is fundamentally incompatible with unrestricted GPU P2P DMA.

NCCL transport layers. NVIDIA's Collective Communications Library (NCCL) supports multiple transport mechanisms: P2P (direct GPU memory access via NVLink or PCIe), SHM (shared memory via CPU), and network (Socket/IP). The default for same-node multi-GPU is P2P, which provides the lowest latency and highest bandwidth. When P2P is unavailable, NCCL can fall back to SHM, which copies data through system memory — slower but functional.

Output Knowledge Created by This Message

This message creates several important outputs:

  1. A confirmed diagnosis. The IO_PAGE_FAULTs provide definitive evidence that the NCCL hang is caused by IOMMU-blocked P2P DMA, not by software configuration errors, SGLang bugs, or PyTorch compatibility issues. This eliminates dozens of possible causes that the assistant had been exploring.
  2. A clear remediation path. Once the root cause is identified, the solution space narrows dramatically: either disable P2P DMA in NCCL (the quick fix) or reconfigure the IOMMU/BIOS to allow P2P under translation (the proper fix). The assistant pursues both paths in subsequent messages.
  3. A documented hardware-software interaction. The message documents that SEV-SNP IOMMU on AMD platforms breaks GPU P2P DMA for Blackwell GPUs connected via PCIe. This is a non-trivial finding that would be valuable for anyone deploying similar configurations.
  4. A pivot point in the debugging strategy. Before this message, the assistant was debugging at the application level (NCCL settings, port configurations, SGLang version). After this message, the debugging shifts to the hardware level (IOMMU configuration, BIOS settings, PCIe capabilities). This represents a fundamental change in approach.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is concise but reveals a clear analytical process. The phrase "Those IO_PAGE_FAULTs from AMD-Vi (IOMMU) on the nvidia GPUs are a big red flag" shows that the assistant immediately recognizes the diagnostic value of the user-provided logs. The use of "big red flag" is not casual — it signals that the assistant understands these faults are not recoverable errors but evidence of a fundamental incompatibility.

The next sentence — "the GPUs are trying to do P2P DMA but IOMMU is blocking it" — demonstrates causal reasoning that connects the observed symptom (IO_PAGE_FAULT) to the mechanism (P2P DMA) to the effect (blocked by IOMMU). This is a three-step causal chain that the assistant traverses in a single sentence.

The third sentence — "This likely happened when the other agent changed IOMMU/vfio settings for the VM passthrough" — shows temporal reasoning that connects the current failure to the configuration change made earlier in the segment. The assistant is constructing a timeline: VM passthrough configuration → IOMMU enabled → P2P DMA broken → NCCL hangs.

Finally, "Let me run some diagnostics" signals the assistant's intent to validate the hypothesis before acting on it. This is methodologically sound — the assistant has formed a strong hypothesis but recognizes the need for confirmation before implementing a fix.

Conclusion

Message <msg id=6193> is a masterclass in diagnostic reasoning. In just a few sentences, the assistant transforms a frustrating, multi-hour debugging session from a dead end into a solved problem. The key insight — that IOMMU page faults from AMD-Vi indicate broken P2P DMA — bridges the gap between the user-provided kernel logs and the NCCL hang that had defied all previous debugging attempts.

This message also illustrates an important principle in systems debugging: the most elusive bugs are often those where the symptom manifests at a different level of abstraction than the root cause. The NCCL hang looked like a software issue (port conflicts, deadlocks, version incompatibility), but its root cause was a hardware virtualization security feature. Only by connecting the kernel-level IO_PAGE_FAULTs to the application-level NCCL behavior could the assistant break the deadlock.

The message's brevity belies its significance. It is the turning point of the entire segment — the moment when confusion gives way to clarity, and when the debugging effort shifts from exploration to remediation.