The Directional P2P Puzzle: Diagnosing Partial IOMMU Remapping on Blackwell GPUs

In the sprawling, multi-session effort to deploy large language models on a cluster of NVIDIA RTX PRO 6000 Blackwell GPUs, few problems proved as stubborn as peer-to-peer (P2P) DMA. The subject message — message 6333 in the conversation — represents a pivotal diagnostic moment. After an exhaustive and ultimately failed attempt to restore full P2P DMA by switching IOMMU groups to identity mode (a technique that crashed the Blackwell Firmware Security Processor), the assistant pivoted to a completely different hypothesis: perhaps the nvidia driver's built-in DmaRemapPeerMmio=1 parameter could create the necessary IOMMU mappings for peer GPU BAR accesses, even under the default DMA-FQ (full translation) IOMMU mode. The test results were in, and message 6333 is where the assistant interprets them, revealing a fascinatingly asymmetric failure pattern that points to a subtle driver-level bug.

The Long Road to This Moment

To understand the significance of message 6333, one must appreciate the journey that led to it. The assistant had been battling with P2P DMA for multiple segments. The core problem was straightforward: with the AMD IOMMU in its default translation mode (DMA-FQ), P2P transfers between GPUs generated IO_PAGE_FAULTs because the IOMMU had no mappings for peer GPU BAR regions. The natural solution was to switch the relevant IOMMU groups to "identity" mode, which bypasses translation and allows direct physical address access. However, this approach ran headlong into a hard wall: Blackwell GPUs require the nvidia driver to be loaded before the IOMMU groups are set to identity, because the GPU's Firmware Security Processor (FSP) boot sequence demands specific DMA mappings set up by the kernel's DMA API in translation mode. Identity mode during FSP initialization causes error code 0x177 and a permanent failure state that no amount of PCI Secondary Bus Reset (SBR) or Function Level Reset (FLR) can clear ([msg 6319], [msg 6320]).

The assistant discovered a working recovery sequence — remove the GPUs from the PCI bus, perform SBR on their upstream bridges, load the nvidia module, then rescan — but this sequence could not accommodate the identity domain switch because nvidia refuses to load without matching PCI devices present ([msg 6323]). The chicken-and-egg problem seemed insurmountable: nvidia needs PCI devices to load, but identity must be set before nvidia probes, and identity cannot be set on non-existent IOMMU groups (after removal) or while nvidia is bound.

The Pivot to DmaRemapPeerMmio

Defeated on the identity-domain front, the assistant shifted strategy. While examining nvidia driver parameters via /proc/driver/nvidia/params, it spotted a critical parameter: DmaRemapPeerMmio=1 ([msg 6327]). This parameter controls whether the nvidia driver explicitly maps peer GPU MMIO (BAR) regions through the IOMMU DMA remapping API. If this mechanism worked correctly, it would solve P2P DMA entirely within the translation-mode IOMMU — no identity domains needed. The assistant wrote a focused P2P transfer test ([msg 6330]), copied it to the inference container ([msg 6331]), and ran it with NCCL_P2P_DISABLE="" to ensure P2P was enabled ([msg 6332]).

The test results were tantalizingly close to success — but not quite there.

Message 6333: Interpreting the Asymmetric Failure

The subject message begins with the assistant's immediate reaction:

Interesting! DmaRemapPeerMmio=1 is partially working. Some pairs transfer correctly and some don't. GPU 0 sending to others fails, but higher-numbered GPUs sending to lower-numbered ones work. This is consistent with the IOMMU not having proper mappings for some peer BAR ranges — specifically, the IOMMU DMA remapping works in one direction but not the other.

This is a classic moment of scientific reasoning in systems debugging. The assistant has a hypothesis (DmaRemapPeerMmio creates IOMMU mappings), a prediction (P2P should work), and an observation (it works for some pairs but not others). The task now is to explain the pattern of failure, because a pattern contains more information than a simple pass/fail result.

The assistant then articulates the exact directional asymmetry:

The pattern: transfers FROM GPU 0 (01:00.0) TO any other GPU fail. FROM GPU 1,2,3 TO GPU 0 works. FROM GPU 1 TO GPU 2,3 fails. FROM GPU 2,3 TO lower-numbered GPUs works. This suggests a directional IOMMU mapping issue — likely the nvidia driver is only mapping BAR ranges for GPUs with lower PCI addresses, or the mapping order matters.

Let us parse this pattern carefully. There are four GPUs, with PCI addresses:

Let me also check dmesg for IO_PAGE_FAULTs during this test: [bash] ssh root@10.1.2.6 'dmesg | grep IO_PAGE_FAULT | tail -20' [85093.872689] nvidia 0000:61:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x73000826240 flags=0x0030] [85093.873006] nvidia 0000:61:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x6e000014000 flags=0x0030] [85093.873141] nvidia 0000:61:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x6e000010000 flags=0x0030] [85093.873266] nvidia 0000:61:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x6e000010100 flags=0x0030]

The IO_PAGE_FAULTs are all logged against GPU 2 (0000:61:00.0), which is the device that initiated the failing DMA transfers. The fault addresses (0x73000826240, 0x6e000014000, etc.) point to peer GPU BAR regions that the IOMMU has no mapping for. This confirms that the nvidia driver's DmaRemapPeerMmio mechanism is indeed creating IOMMU mappings — but only for a subset of peer relationships. The mappings that exist are sufficient for some directions but not others.

The Reasoning Process

What makes message 6333 so interesting is the quality of the reasoning on display. The assistant is not simply reporting test results; it is actively interpreting them through the lens of systems knowledge. Several layers of inference are at work:

First, the assistant recognizes that partial success is more informative than total failure. If all P2P transfers had failed, the conclusion would be "DmaRemapPeerMmio doesn't work on this driver version." If all had succeeded, the conclusion would be "the problem is solved." Instead, the asymmetric pattern tells a more nuanced story: the mechanism is operating but incompletely. This narrows the search space considerably.

Second, the assistant maps the behavioral pattern to a structural hypothesis. The directional asymmetry — transfers to lower-index GPUs succeed while transfers to higher-index GPUs fail — is not random. It suggests an ordering dependency in how the driver creates IOMMU mappings. The assistant proposes two mechanisms: "the nvidia driver is only mapping BAR ranges for GPUs with lower PCI addresses, or the mapping order matters." Both explanations are consistent with the observed data and are testable.

Third, the assistant independently verifies the failure mechanism. Rather than relying solely on the Python test output, it cross-references with kernel log messages. The IO_PAGE_FAULTs in dmesg confirm that the IOMMU is indeed the point of failure — the DMA engine on GPU 2 is attempting to access peer BAR addresses that have no valid IOMMU mapping. This triangulation between application-level test results and kernel-level error logs is a hallmark of rigorous debugging.

Assumptions and Input Knowledge

To fully understand message 6333, several pieces of background knowledge are essential. The reader must understand what P2P DMA is in the context of multi-GPU computing — the ability for one GPU to directly read or write the memory of another GPU across the PCIe fabric without going through the CPU or system memory. They must understand the role of the IOMMU (Input-Output Memory Management Unit) in translating device DMA addresses to physical memory addresses, and the distinction between translation mode (DMA-FQ, where all DMA addresses go through the IOMMU) and identity mode (where DMA addresses pass through untranslated).

The reader must also understand the nvidia driver architecture: the DmaRemapPeerMmio parameter is a driver-level knob that instructs the nvidia kernel module to create explicit IOMMU mappings for peer GPU BAR (Base Address Register) regions. Without these mappings, a GPU attempting to DMA to another GPU's BAR will trigger an IO_PAGE_FAULT because the IOMMU has no translation entry for that address range.

The assistant makes a key assumption in this message: that the directional asymmetry is caused by a bug in the nvidia driver's mapping logic rather than a hardware limitation or a PCIe topology constraint. This is a reasonable assumption given the pattern, but it is not yet proven. The assistant also assumes that the IO_PAGE_FAULTs are the cause of the transfer failures rather than a symptom of some deeper issue — again reasonable, but worth noting.

Output Knowledge Created

Message 6333 produces several valuable pieces of output knowledge:

  1. DmaRemapPeerMmio=1 is partially functional on the nvidia driver version 590.48.01 with Blackwell GPUs under AMD IOMMU DMA-FQ mode. It creates IOMMU mappings for some peer BAR ranges but not all.
  2. The failure pattern is directional and asymmetric, strongly suggesting an ordering bug in the driver's mapping creation logic. The hypothesis that "the nvidia driver is only mapping BAR ranges for GPUs with lower PCI addresses" is the leading theory.
  3. The IO_PAGE_FAULTs confirm the IOMMU as the point of failure, with the faulting device being the initiating GPU (GPU 2 in the logged examples) and the fault addresses corresponding to peer BAR regions.
  4. The identity-domain approach is definitively blocked for Blackwell GPUs, making DmaRemapPeerMmio the only viable path to P2P DMA under full IOMMU translation.

What Comes Next

The message ends with the assistant gathering more data (dmesg output) to confirm the IOMMU fault mechanism. This sets the stage for the next phase of debugging: understanding why the nvidia driver creates incomplete mappings and whether there is a workaround. The partial success of DmaRemapPeerMmio is both encouraging and frustrating — it proves the mechanism can work, but reveals a bug that prevents it from working fully. The assistant's next steps would logically involve examining the nvidia driver source (if available), testing with different GPU ordering or PCIe topology configurations, or exploring whether a kernel boot parameter like iommu.passthrough=1 could provide a system-wide solution without triggering the Blackwell FSP boot failure.

In the broader arc of the conversation, message 6333 represents a moment of clarity after a long period of trial and error. The assistant has ruled out the identity-domain approach, discovered a partially working alternative, and characterized its failure mode with enough precision to guide further investigation. It is a textbook example of systematic debugging: form a hypothesis, test it, interpret the results, and use the pattern of failure to refine the next hypothesis.