When IOMMU Breaks GPU P2P DMA: Diagnosing a Blackwell Deployment Under Full Translation

Introduction

In the high-stakes world of large-scale ML inference deployment, the boundary between hardware configuration and software workaround is often where the most interesting engineering happens. Message 6239 of this opencode session captures exactly such a boundary moment: a user asks whether BIOS-level settings can restore GPU peer-to-peer (P2P) DMA functionality that was broken by a virtualization security feature, and the assistant responds with a concise technical explanation followed by a targeted investigation command. Though brief in appearance, this message represents a critical pivot point in a multi-session optimization journey spanning CUDA upgrades, NCCL tuning, and Blackwell GPU topology reconfiguration.

The Context: A P2P DMA Crisis

The conversation leading up to this message had been a rollercoaster of deployment and debugging. The team was running Qwen3.5-122B-A10B BF16 across 4× RTX PRO 6000 Blackwell GPUs on a Proxmox host, using SGLang as the inference engine. Earlier in segment 40, the assistant had reconfigured the GPU topology, splitting 8 Blackwell GPUs between an LXC container and a SEV-SNP VM. This reconfiguration had introduced a critical problem: every GPU-to-GPU P2P DMA transfer produced corrupted data, causing NCCL to hang during distributed initialization.

The root cause had been meticulously diagnosed in the preceding messages ([msg 6211] through [msg 6220]). The assistant had written a CUDA diagnostic script, tested P2P accessibility (which the driver reported as available), then performed actual P2P data transfers and found every single one produced mismatched results. The culprit was the IOMMU (Input-Output Memory Management Unit) running in full translation mode — a consequence of the SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) configuration on the AMD host. The fix had been to disable NCCL P2P entirely via NCCL_P2P_DISABLE=1, forcing NCCL to fall back to shared memory (SHM) transport. This workaround had successfully brought the server online, achieving 108 tok/s single-stream and up to 2,800 tok/s at high concurrency.

But the user was not satisfied with a software workaround. In [msg 6238], they asked: "Is there anything that can be done to get p2p going with iommu? Any bios settings about iommu coverage, related settings?" This question frames the central tension of message 6239: can the hardware configuration be adjusted to restore native P2P performance, or is the software workaround the permanent solution?

The Message: A Concise Technical Diagnosis

Here is the complete message as it appears in the conversation:

Good question. The IO_PAGE_FAULTs happen because with amd_iommu=on (full translation, not passthrough), each GPU's IOMMU domain only has mappings for system memory, not for other GPUs' BARs. When GPU A tries to DMA directly to GPU B's BAR address, the IOMMU has no mapping for it and faults.

>

Let me research the options: `` [bash] ssh root@10.1.2.6 'cat /proc/cmdline; echo "---"; dmesg | grep -i "iommu\|AMD-Vi" | grep -iv page_fault | head -20' initrd=\EFI\proxmox\6.14.11-5-bpo12-pve\initrd.img-6.14.11-5-bpo12-pve root=ZFS=rpool/ROOT/pve-1 amd_iommu=on amd_pstate=active processor.max_cstate=0 nmi_watchdog=0 mem_encrypt=on kvm_amd.sev=1 pcie_aspm=off pci=realloc=off numa_balancing=disable boot=zfs --- ``

The message is structured in two distinct parts. First, a clear technical explanation of why the IO_PAGE_FAULTs occur. Second, an investigative action — the assistant SSHes into the Proxmox host to inspect the kernel boot parameters and IOMMU-related kernel messages.

The Technical Explanation: IOMMU Domains and GPU BARs

The assistant's explanation is deceptively simple but captures a nuanced hardware interaction. When amd_iommu=on is set without the complementary iommu=pt (passthrough) parameter, the IOMMU operates in full translation mode. In this mode, every DMA transaction from a PCIe device must go through address translation. The IOMMU maintains per-device (or per-group) domain tables that map device-virtual addresses to physical system memory addresses.

The critical detail is what these domain tables don't contain. Each GPU in a multi-GPU system has a PCIe Base Address Register (BAR) — a memory-mapped region that allows other devices to access the GPU's framebuffer directly. In full translation mode, the IOMMU domain for GPU A only contains mappings for system RAM, not for the BAR regions of GPU B, GPU C, or GPU D. When GPU A initiates a P2P DMA write to GPU B's BAR address, the IOMMU intercepts the transaction, looks up the address in GPU A's domain table, finds no valid mapping, and raises an IO_PAGE_FAULT. The data transfer completes — but with corrupted data, because the IOMMU either blocks the write or routes it to an incorrect physical location.

This explains the earlier diagnostic results where torch.cuda.can_device_access_peer() returned True for all GPU pairs (the driver-level capability check passes because the hardware can do P2P), but every actual data transfer produced mismatches (the IOMMU corrupts the data in flight). The assistant's explanation correctly identifies the gap between capability and reality — a distinction that had consumed significant debugging effort in the preceding messages.

The Investigation: Reading the Kernel Command Line

The bash command executed in this message serves a specific purpose: to determine exactly how the IOMMU and related virtualization features are configured on the Proxmox host. The command runs two operations: reading /proc/cmdline to see the kernel boot parameters, and filtering dmesg for IOMMU and AMD-Vi messages (excluding the noise of page faults that had been flooding the log).

The output reveals the kernel command line:

initrd=\EFI\proxmox\6.14.11-5-bpo12-pve\initrd.img-6.14.11-5-bpo12-pve root=ZFS=rpool/ROOT/pve-1 amd_iommu=on amd_pstate=active processor.max_cstate=0 nmi_watchdog=0 mem_encrypt=on kvm_amd.sev=1 pcie_aspm=off pci=realloc=off numa_balancing=disable boot=zfs

Several parameters are immediately relevant. amd_iommu=on confirms IOMMU is enabled in full translation mode — there is no iommu=pt parameter that would enable passthrough mode. mem_encrypt=on enables AMD memory encryption (SME), and kvm_amd.sev=1 enables SEV (Secure Encrypted Virtualization) support. The combination of full IOMMU translation with SEV-SNP is precisely what breaks P2P DMA: SEV-SNP requires the IOMMU to enforce memory encryption and integrity, which means every DMA transaction must go through the IOMMU for protection, and the IOMMU cannot safely map one GPU's BAR into another GPU's domain without compromising the security model.

The dmesg portion of the command produced no visible output after the separator line. This could mean that the grep -iv page_fault filter removed all matching lines (since the IO_PAGE_FAULTs had been flooding the log), or that the relevant IOMMU initialization messages were not captured within the head -20 limit. Either way, the kernel cmdline alone provides enough information to understand the configuration.

Assumptions and Input Knowledge

This message makes several implicit assumptions about the reader's technical background. It assumes familiarity with IOMMU concepts — domain tables, full translation vs. passthrough mode, and how DMA transactions interact with the IOMMU. It assumes understanding of GPU BARs and their role in P2P communication. It assumes knowledge of the SEV-SNP security architecture and why it necessitates full IOMMU translation. It also assumes familiarity with the Proxmox virtualization platform and its kernel parameter conventions.

The input knowledge required to fully understand this message includes: the earlier diagnostic work showing that P2P transfers produce corrupted data (not just failures); the understanding that canAccessPeer=True from the CUDA runtime indicates driver-level capability, not actual functional P2P; and the context that SEV-SNP was enabled for confidential computing use cases that require memory encryption and integrity protection.

The assistant also makes an operational assumption: that the Proxmox host is accessible via SSH at the specified IP address and that the kernel command line and dmesg output will provide actionable information. This is a reasonable assumption given that the assistant had been managing this host throughout the session.

Output Knowledge and Significance

The primary output of this message is the confirmed kernel configuration. The cmdline reveals that amd_iommu=on is set without iommu=pt, and that SEV-SNP is enabled. This combination means that restoring P2P DMA is not simply a matter of adding a BIOS setting — it would require either:

  1. Disabling full IOMMU translation by adding iommu=pt to the kernel cmdline, which would put the IOMMU into passthrough mode for most devices. However, this may conflict with SEV-SNP requirements, as SEV-SNP typically requires IOMMU enforcement for integrity.
  2. Selectively enabling P2P within IOMMU domains, which would require specific IOMMU hardware support for inter-device mappings — a feature that may not be available or may be deliberately disabled under SEV-SNP.
  3. Rebooting with modified kernel parameters to test passthrough mode, accepting the potential security trade-off. The message also produces a secondary output: the dmesg filter command structure. By excluding page fault messages (grep -iv page_fault), the assistant demonstrates a diagnostic technique for cutting through noise to find relevant IOMMU configuration messages.

The Thinking Process

The assistant's reasoning in this message follows a clear pattern. First, it validates the user's question as "good" — acknowledging that BIOS-level investigation is the right next step after exhausting software workarounds. Then it provides the mechanistic explanation, which serves both to educate and to frame the investigation: if we understand why P2P fails, we know what to look for in the configuration. The explanation identifies the specific condition — "each GPU's IOMMU domain only has mappings for system memory, not for other GPUs' BARs" — which directly implies that the solution must involve either expanding IOMMU domains to include peer BARs or reducing IOMMU enforcement.

The decision to check /proc/cmdline first is strategic: kernel boot parameters are the software-visible representation of BIOS and firmware configuration. They reveal what the hardware was told to do at boot time, which is the necessary first step before exploring BIOS menu options. The dmesg filter complements this by showing how the IOMMU driver interpreted those parameters during initialization.

Broader Significance

This message sits at the intersection of two trends in modern ML infrastructure: the push toward confidential computing (which requires IOMMU enforcement and memory encryption) and the demand for maximum GPU performance (which requires unfettered P2P DMA). The tension between security and performance is not new, but it takes a specific form in the Blackwell/SEV-SNP context. The assistant's investigation in this message sets the stage for the next round of decision-making: whether to accept the NCCL_P2P_DISABLE performance penalty, or to reconfigure the virtualization layer to restore P2P at the cost of reduced security isolation.

The message also exemplifies a broader engineering principle: when a software workaround exists (NCCL_P2P_DISABLE), the question of whether to pursue a hardware fix depends on the performance impact. With the workaround, the system achieves 2,800 tok/s at C=128. The question is whether P2P DMA would significantly improve that number — and whether the improvement justifies the configuration complexity and potential security regression. That calculation is left for future messages, but message 6239 provides the data needed to make it.