The IO_PAGE_FAULT Evidence: Diagnosing GPU P2P DMA Corruption Under IOMMU Full Translation

Message Overview

The subject message ([msg 6241]) is a single bash command executed by the assistant on a Proxmox virtualization host:

ssh root@10.1.2.6 'dmesg | grep -iE "iommu|AMD.Vi" | head -30'

Its output reveals a stream of IO_PAGE_FAULT events logged by the AMD-Vi IOMMU for NVIDIA RTX PRO 6000 Blackwell GPUs:

[71667.128418] nvidia 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x29000826200 flags=0x0030]
[71667.128532] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x39000826240 flags=0x0030]
[71667.128552] nvidia 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x24000000000 flags=0x0030]
[71667.128662] nvidia 0000:11:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x003f address=0x34000011000 flags=0x0030]
[71667.12870...

On the surface, this is a trivial diagnostic command — just grepping kernel logs. But in the broader context of the session, this message represents a critical pivot point: the moment the assistant transitions from working around the IOMMU P2P DMA problem to understanding its root cause in response to a direct question from the user. The message is deceptively simple, yet it carries the weight of an entire diagnostic investigation that had been building across dozens of prior messages.

The Context That Produced This Message

To understand why this message exists, we must trace the chain of events that led to it. The session had been running on a Proxmox host (kpro6) with 8× NVIDIA RTX PRO 6000 Blackwell GPUs, split 4+4 between an LXC container running SGLang inference and a SEV-SNP confidential VM managed by another tenant. The SEV-SNP (Secure Encrypted Virtualization — Secure Nested Paging) configuration required the kernel to boot with amd_iommu=on in full translation mode rather than the more permissive iommu=pt (passthrough) mode. This seemingly innocuous kernel parameter had catastrophic consequences for GPU-to-GPU communication.

Earlier in the session ([msg 6216]), the assistant had discovered that every GPU-to-GPU P2P DMA transfer produced corrupted data. The CUDA driver reported can_device_access_peer=True — it believed P2P was functional — but actual data transfers between any pair of GPUs resulted in mismatches. The assistant had already diagnosed this as an IOMMU issue ([msg 6217]) and implemented a workaround by setting NCCL_P2P_DISABLE=1, forcing NCCL to use shared memory (SHM) transport instead of P2P DMA. This workaround was successful — the SGLang server loaded and achieved up to 2,800 tok/s at high concurrency ([msg 6231]).

However, the user was not satisfied with a workaround. In [msg 6238], they asked: "Is there anything that can be done to get p2p going with iommu? Any bios settings about iommu coverage, related settings?" This question reframed the problem: instead of accepting the P2P disable as a permanent fix, the user wanted to explore whether the root cause could be addressed, potentially restoring the performance benefits of GPU-to-GPU direct DMA.

The assistant's response in [msg 6239] began investigating. It first checked the kernel command line (cat /proc/cmdline) to confirm the IOMMU configuration, and ran dmesg | grep -i "iommu|AMD-Vi" | grep -iv page_fault to look for non-fault IOMMU messages. That command returned no output — all IOMMU-related kernel messages were page faults. Then in [msg 6240], the assistant ran the same command again but with a slightly different pattern, still filtering out page faults. Again, empty output.

The subject message ([msg 6241]) is the third attempt. This time, the assistant removes the -v page_fault filter and uses a refined regex pattern (-iE "iommu|AMD.Vi"). The result is a flood of IO_PAGE_FAULT events — the concrete evidence the assistant needed.

Why This Message Matters

This message is the diagnostic breakthrough. The IO_PAGE_FAULT entries in the kernel log are not just error messages; they are the fingerprints of the underlying hardware interaction failure. Each fault record contains:

The PCI device: nvidia 0000:01:00.0 and nvidia 0000:11:00.0 identify specific GPU instances by their PCI bus address.
The IOMMU domain: domain=0x002c and domain=0x003f indicate which IOMMU translation domain the faulting device belongs to. Each GPU in its own domain confirms that ACS (Access Control Services) is enabled, isolating each device.
The faulting address: Addresses like 0x29000826200 and 0x24000000000 are GPU BAR (Base Address Register) addresses — the memory-mapped regions of peer GPUs. This is the critical clue: GPU A is trying to DMA directly to GPU B's BAR, and the IOMMU has no valid translation for that address.
The flags: flags=0x0030 indicates the type of fault (typically a write access to a non-present page). The assistant's understanding of these fault records is evident from the analysis in [msg 6239]: "each GPU's IOMMU domain only has mappings for system memory, not for other GPUs' BARs. When GPU A tries to DMA directly to GPU B's BAR address, the IOMMU has no mapping for it and faults." This interpretation is correct and reflects deep knowledge of how IOMMU translation works in virtualized environments.

The Thinking Process Visible in the Message Sequence

The progression from [msg 6239] through [msg 6241] reveals a methodical diagnostic approach:

Hypothesis formation: The assistant correctly hypothesized that IO_PAGE_FAULTs were the root cause of P2P corruption, based on earlier observations of NCCL hangs and data mismatches.
Evidence gathering — attempt 1 ([msg 6239]): The assistant ran dmesg | grep -i "iommu|AMD-Vi" | grep -iv page_fault to look for non-fault IOMMU messages that might reveal configuration details. This returned only the kernel cmdline output (from the first part of the command), but the dmesg grep itself produced no output.
Evidence gathering — attempt 2 ([msg 6240]): The assistant tried again with the same filter, still getting empty output. At this point, the assistant realized that all IOMMU messages in the log were page faults — there were no informational messages about IOMMU initialization or configuration.
Evidence gathering — attempt 3 ([msg 6241]): The assistant removed the page fault filter and used a broader regex. This immediately produced the expected fault records, confirming the hypothesis. This iterative refinement is characteristic of good diagnostic work: each attempt narrows the search space based on what was learned from the previous attempt. The assistant could have jumped straight to the unfiltered grep, but the sequential approach demonstrates a systematic methodology — first check for non-fault messages (which might contain useful configuration info), and only when those are absent, examine the faults themselves.

Assumptions and Their Validity

The assistant operated under several assumptions in this message:

Assumption 1: The IO_PAGE_FAULTs are the direct cause of P2P data corruption. This is well-supported by the evidence. The fault addresses correspond to GPU BAR regions, and the corruption pattern (every GPU-to-GPU transfer fails) matches what one would expect when the IOMMU blocks DMA writes to peer BARs. The assistant had already confirmed that NCCL_P2P_DISABLE=1 resolves the issue, which is consistent with this interpretation.

Assumption 2: Examining the fault records will help find a solution. This is reasonable but not yet validated at this point in the conversation. The fault records confirm the mechanism of failure, but they don't immediately suggest a fix. The subsequent investigation ([msg 6242] onward) explores IOMMU group types, per-group identity domains, and kernel parameters — all informed by the fault evidence gathered here.

Assumption 3: The head -30 limit is sufficient. The assistant truncated output to 30 lines. Given that the faults are repetitive (same pattern for different GPU pairs), this is a reasonable assumption — the first few entries contain the essential information.

Input Knowledge Required

To fully understand this message, one needs:

IOMMU architecture: Understanding that an IOMMU (I/O Memory Management Unit) translates device DMA addresses to physical memory addresses, similar to how a CPU's MMU translates virtual addresses. In full translation mode, all device DMA must go through the IOMMU.
AMD-Vi: AMD's implementation of IOMMU, part of the AMD-V virtualization technology. The "Vi" stands for "Virtualization Interface."
IO_PAGE_FAULT semantics: A page fault from the IOMMU occurs when a device attempts to DMA to an address that has no valid mapping in its IOMMU domain. The flags field encodes the type of access (read/write) and the reason for the fault.
GPU BARs: PCIe devices expose memory-mapped regions called BARs (Base Address Registers). GPUs use BARs for VRAM access and inter-GPU communication. P2P DMA works by one GPU writing directly to another GPU's BAR.
PCIe topology and ACS: Access Control Services (ACS) is a PCIe feature that enables IOMMU groups to be separated per device. With ACS enabled, each GPU gets its own IOMMU domain, preventing direct DMA between devices without IOMMU translation.
The session's hardware context: 8× RTX PRO 6000 Blackwell GPUs on an AMD platform, split between LXC and VM, with SEV-SNP requiring full IOMMU translation.

Output Knowledge Created

This message produced several concrete outputs:

Confirmed evidence of IO_PAGE_FAULTs: The kernel log entries are now captured in the conversation, providing verifiable proof that the IOMMU is blocking GPU-to-GPU DMA.
Device identification: The PCI addresses (0000:01:00.0, 0000:11:00.0) identify which specific GPUs are faulting. These correspond to the NUMA 0 GPUs assigned to the LXC container.
Address pattern analysis: The faulting addresses (e.g., 0x24000000000, 0x34000011000) are in the GPU BAR range, confirming the P2P nature of the failing transfers.
Domain separation confirmation: Different GPUs have different domain IDs (0x002c vs 0x003f), confirming ACS is enabled and each GPU is in its own IOMMU group.
Baseline for solution exploration: The fault evidence provides the foundation for the subsequent investigation into per-group identity domains, kernel parameters, and BIOS settings that follows in later messages.

Broader Significance

This message exemplifies a pattern that recurs throughout the opencode session: the assistant uses precise, targeted diagnostic commands to convert observed symptoms into concrete evidence, then uses that evidence to drive decision-making. The IO_PAGE_FAULT discovery is not an end in itself — it is the necessary precondition for exploring solutions. Without this evidence, the assistant would be speculating about the cause; with it, the assistant can reason about specific mitigation strategies.

The message also highlights the tension between security features and performance in virtualized GPU environments. SEV-SNP requires full IOMMU translation to protect VM memory from device DMA attacks, but this protection comes at the cost of breaking GPU P2P DMA — a feature that can significantly improve multi-GPU inference performance. The user's question about restoring P2P reflects a desire to have both security and performance, a challenge that has no easy answer in the current hardware/software ecosystem.

In the messages that follow ([msg 6242] onward), the assistant uses the fault evidence to investigate IOMMU group types, per-group identity domains, and NVIDIA driver parameters — all aimed at finding a way to exempt the inference GPUs from translation while keeping the VM GPUs protected. Whether such a selective configuration is possible remains an open question at this point in the session, but the diagnostic foundation laid in this message makes that investigation possible.