The Discovery of DmaRemapPeerMmio: A Pivot Point in the Blackwell P2P DMA Saga

In the long and arduous journey to enable peer-to-peer (P2P) DMA communication between NVIDIA Blackwell GPUs under a virtualized IOMMU configuration, message [msg 6327] marks a quiet but pivotal moment. After dozens of failed attempts to force the system into IOMMU identity (passthrough) mode — attempts that culminated in the discovery that Blackwell's Firmware Security Processor (FSP) fundamentally cannot boot when the IOMMU is in identity mode — the assistant takes a step back, reads the nvidia driver's runtime parameters, and discovers a potentially crucial built-in mechanism: DmaRemapPeerMmio. This message captures the moment of realization that the solution to the P2P puzzle might already exist within the nvidia driver itself, waiting to be understood rather than hacked in from outside.

The Context: A Battle Against Blackwell's Firmware

To understand the weight of this message, one must appreciate the ordeal that preceded it. The assistant had been working on a system with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs on NUMA0, bound to the nvidia driver) and a VM (4 GPUs on NUMA1, bound to vfio-pci for SEV-SNP). The core problem was straightforward in concept but maddening in practice: P2P DMA between the NUMA0 GPUs was broken under the default IOMMU translation mode (DMA-FQ), causing IO_PAGE_FAULT errors when GPUs attempted to access each other's BAR (Base Address Register) memory regions.

The obvious fix — setting the IOMMU groups to identity mode (passthrough, which bypasses DMA remapping entirely) — had proven catastrophically incompatible with Blackwell. The nvidia driver's GSP (GPU System Processor) / FSP firmware boot sequence requires specific DMA mappings that are only set up by the kernel's DMA API in translation mode. When identity mode was enabled, the FSP boot failed with error code 0x177, and no amount of Secondary Bus Reset (SBR), Function-Level Reset (FLR), or PCIe hot-remove/rescan could clear the corrupted firmware state. The assistant had spent message after message — spanning dozens of iterations — trying every conceivable timing variation: setting identity before the nvidia driver loaded, setting identity after, using modprobe install hooks, attempting to unbind and rebind, using driver_override, and even exploring kernel boot parameters like iommu.passthrough=1. Every path was blocked.

The only working configuration that had been recovered was the default DMA-FQ mode, with P2P DMA disabled via NCCL_P2P_DISABLE=1. The system was stable and serving Qwen3.5-122B-A10B BF16 at impressive throughput thanks to MTP speculation, but the P2P problem remained unsolved.

The Message: Reading the Nvidia Driver's Secrets

[msg 6327] begins with the assistant examining the output of /proc/driver/nvidia/params, a file that exposes the nvidia kernel module's runtime configuration parameters. The assistant had just read this file in the previous message ([msg 6326]) and is now reacting to what it found:

DmaRemapPeerMmio: 1 — This is extremely interesting! This parameter controls whether the nvidia driver maps peer GPU MMIO (BAR) regions through the IOMMU DMA remapping. It's already enabled (value 1).

>

And EnablePCIERelaxedOrderingMode: 0 and EnableResizableBar: 0 — both disabled.

>

Let me look deeper at DmaRemapPeerMmio. If it's set to 1, the driver should be creating IOMMU mappings for peer GPU BARs, which would solve the P2P DMA issue under full IOMMU translation. But we're still seeing IO_PAGE_FAULTs... Let me check if this parameter was working or if there's a bug.

>

Actually, let me just test P2P right now with the current working state (DMA-FQ mode, DmaRemapPeerMmio=1) — maybe the driver has fixed this in 590.48:

The assistant then runs a simple nvidia-smi command to confirm the GPUs are alive and responsive, receiving back the expected output showing all four NUMA0 GPUs:

0, 00000000:01:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition
1, 00000000:11:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition
2, 00000000:61:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition
3, 00000000:71:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition

The Reasoning: Connecting the Dots

What makes this message significant is the chain of reasoning it reveals. The assistant has been operating under the assumption that IOMMU identity mode was the only way to enable P2P DMA — because identity mode bypasses DMA translation entirely, allowing GPUs to directly address each other's physical BAR addresses without the IOMMU intercepting and potentially faulting on those transactions. The entire multi-session effort had been directed at making identity mode work.

But now, reading DmaRemapPeerMmio=1, the assistant realizes there is a second path. The nvidia driver itself has a mechanism for creating proper IOMMU mappings for peer GPU BARs. When this parameter is enabled, the driver should, during its initialization, call into the kernel's DMA API to map each peer GPU's BAR region into the IOMMU page tables of the requesting GPU. This would allow P2P DMA to function correctly even under full IOMMU translation mode — the IOMMU would have valid mappings for peer BAR accesses and would translate them correctly instead of raising IO_PAGE_FAULTs.

The assistant's thinking is visible in the careful phrasing: "If it's set to 1, the driver should be creating IOMMU mappings for peer GPU BARs, which would solve the P2P DMA issue under full IOMMU translation. But we're still seeing IO_PAGE_FAULTs..." This reveals a tension between what the parameter should do and what the system is actually experiencing. The assistant correctly identifies a discrepancy and formulates a testable hypothesis: perhaps the driver version 590.48 has fixed a bug that existed in earlier versions, or perhaps the parameter was not actually taking effect for some other reason.

The decision to "test P2P right now" is a pragmatic one. Rather than diving into speculative analysis of the driver source code or kernel internals, the assistant chooses to run an empirical test. The nvidia-smi command serves as a quick sanity check — confirming the GPUs are present and the driver is loaded — before proceeding to a more targeted P2P bandwidth test.

Assumptions and Their Risks

The message rests on several implicit assumptions that deserve examination. First, the assistant assumes that DmaRemapPeerMmio=1 is a functional, complete solution — that enabling this parameter should, in theory, make P2P work under DMA-FQ mode. This is a reasonable assumption given the parameter's documented purpose, but it may be incomplete. The nvidia driver's peer MMIO remapping might have limitations: it might only work for certain GPU pairs, or it might require specific PCIe topology configurations, or it might interact poorly with the Resizable BAR feature (which is disabled, as noted).

Second, the assistant assumes that the current driver version (590.48.01) might have fixed whatever bug caused P2P to fail previously. This is an optimistic assumption — driver updates do fix bugs, but the assistant has no evidence that this specific issue was addressed. The test will confirm or refute this.

Third, there is an implicit assumption that the DmaRemapPeerMmio mechanism, if it works, would be a complete replacement for IOMMU identity mode. In reality, even if the driver creates correct IOMMU mappings, there might be performance implications — going through the IOMMU for P2P transactions adds latency compared to direct physical addressing. The assistant's immediate goal, however, is functionality rather than peak performance.

The Broader Narrative Arc

This message sits at a transition point in the larger story. The IOMMU identity domain approach has been definitively ruled out — the Blackwell FSP incompatibility is a hard constraint that no amount of software manipulation can overcome. The assistant has been casting about for alternatives, and this discovery offers a new direction.

What makes the moment compelling is the contrast between the failed complexity of the IOMMU approach and the elegant simplicity of the potential solution. The IOMMU identity path required intricate sequences of PCIe hot-remove, SBR timing, modprobe hooks, and kernel parameter manipulation — all of which ultimately failed against Blackwell's firmware requirements. The DmaRemapPeerMmio path, by contrast, is a single parameter that the nvidia driver already exposes and already has set to 1. The question is not how to enable it, but why it isn't working as documented.

The message also reveals the assistant's methodology: when a theoretical approach fails, pivot to empirical testing. Rather than continuing to speculate about why DmaRemapPeerMmio might not be working, the assistant decides to simply test P2P and observe the result. This empirical mindset has been a consistent thread throughout the session — each hypothesis is tested, each failure produces data that informs the next attempt.

Input and Output Knowledge

To fully understand this message, the reader needs several pieces of input knowledge. First, an understanding of IOMMU concepts — what DMA remapping does, what identity/passthrough mode means, and how IO_PAGE_FAULTs occur when the IOMMU lacks mappings for a transaction. Second, familiarity with the nvidia driver's parameter system and the /proc/driver/nvidia/params interface. Third, awareness of the preceding struggle with Blackwell's FSP boot failure in identity mode. Fourth, knowledge of P2P DMA in multi-GPU systems — how GPUs communicate directly over PCIe without host memory involvement.

The output knowledge created by this message is more subtle. The immediate output is confirmation that the GPUs are operational — the nvidia-smi output shows all four devices. But the more important output is the identification of DmaRemapPeerMmio as a potential path forward, and the framing of a testable hypothesis: "maybe the driver has fixed this in 590.48." This hypothesis will drive the next phase of investigation, leading to P2P bandwidth tests that will reveal whether the driver's built-in remapping actually works.

Conclusion

Message [msg 6327] is a moment of intellectual pivot in a long debugging session. After the IOMMU identity approach was shattered against the immovable object of Blackwell's firmware requirements, the assistant discovers that the nvidia driver may already contain the solution. The DmaRemapPeerMmio parameter represents a completely different strategy: instead of bypassing the IOMMU, work with it by having the driver create the necessary mappings. Whether this path will succeed remains to be seen — the chunk summary hints that the mappings are "incomplete, with some peer pairs working and others faulting" — but the pivot itself is a textbook example of adaptive problem-solving. When one approach hits a hard wall, look for the tool that's already in your hand.