The Pivot: From IOMMU Identity Domains to Nvidia's DmaRemapPeerMmio

In the course of a single read tool call, an entire strategy shifts. Message [msg 6329] appears, at first glance, to be the most mundane of operations: an AI assistant reading lines 30 through 41 of a Python file called gpu_diag.py. The content it retrieves is a fragment of a GPU diagnostics script — a pair of nested loops that check peer-to-peer (P2P) access capability between GPUs using torch.cuda.can_device_access_peer(). Nothing more. Yet this simple act of reading code is the fulcrum upon which a much larger investigation pivots, marking the transition from one failed approach to a new, more promising line of inquiry.

The Context of Failure

To understand why this message matters, one must understand what came before it. The assistant had been engaged in a multi-hour battle to enable P2P DMA (direct memory access between GPUs) on a system with four NVIDIA RTX PRO 6000 Blackwell GPUs, running under an AMD IOMMU with SEV-SNP (Secure Encrypted Virtualization — Secure Nested Paging) enabled. P2P DMA is critical for multi-GPU machine learning workloads — it allows GPUs to directly read each other's memory without going through the CPU, dramatically accelerating distributed tensor operations, NCCL all-reduce, and model parallelism.

The original approach was elegant in theory: set the IOMMU domain type to identity (passthrough mode) for the GPU groups, which would bypass IOMMU translation for those devices and allow direct PCIe peer access. The assistant had spent many messages developing a sophisticated procedure: unload the nvidia driver, set IOMMU groups to identity, remove the GPU devices from the PCI bus, perform a Secondary Bus Reset (SBR) on the upstream bridges, reload the nvidia driver, and rescan the bus. This should have given the driver fresh, clean devices with identity IOMMU domains.

But reality intervened. The Blackwell GPUs' Firmware Security Processor (FSP) — a dedicated security co-processor that handles GPU initialization, attestation, and secure boot — failed with error code 0x177 whenever the IOMMU was in identity mode. The FSP, it turned out, requires specific DMA mappings set up by the kernel's DMA API during initialization. Identity mode bypasses these mappings, and the FSP cannot complete its boot sequence. This was not a recoverable error: no amount of Secondary Bus Reset, Function-Level Reset (FLR), or even CXL bus reset could clear the corrupted FSP state once it had occurred. The assistant had to revert, restoring the GPUs to DMA-FQ (DMA with Flush Queue) mode, the kernel's default translated IOMMU domain type.

A New Lead: DmaRemapPeerMmio

After restoring the GPUs to working order, the assistant took stock. In [msg 6326] and [msg 6327], it examined the nvidia driver's module parameters by reading /proc/driver/nvidia/params. There, among dozens of parameters, one stood out: DmaRemapPeerMmio: 1.

This parameter controls whether the nvidia driver creates IOMMU DMA mappings for peer GPU Memory-Mapped I/O (MMIO) regions — the BAR (Base Address Register) spaces that GPUs use for P2P access. When set to 1, the driver should, in theory, use dma_map_resource() or similar kernel APIs to map peer GPU BARs through the IOMMU, enabling P2P DMA even under full IOMMU translation. This was potentially the solution the assistant had been seeking all along — a way to get P2P working without touching IOMMU domain types at all.

The assistant's reasoning, visible in [msg 6327], reveals its thought process: "If it's set to 1, the driver should be creating IOMMU mappings for peer GPU BARs, which would solve the P2P DMA issue under full IOMMU translation. But we're still seeing IO_PAGE_FAULTs... Let me check if this parameter was working or if there's a bug."

This is the critical moment. The assistant had been operating under the assumption that P2P DMA required identity IOMMU domains — an assumption that had now been proven both difficult and dangerous (capable of bricking GPUs). But the nvidia driver itself might already have a mechanism for P2P under translation. The question was: does it actually work?

Message 6329: Reading the Diagnostic Code

Message [msg 6329] is the assistant's preparation to answer that question. Having just confirmed in [msg 6328] that the container sees all four GPUs, the assistant decides to run a P2P DMA test. But first, it needs to look at the existing diagnostic code to understand what tests are already available and what needs to be written.

The read tool call retrieves lines 30-41 of gpu_diag.py:

30:             if i == j:
31:                 continue
32:             can = torch.cuda.can_device_access_peer(i, j)
33:             print(f"  GPU {i} -> GPU {j}: canAccessPeer={can}")
34: 
35:     print("\n=== P2P Transfer Tests ===")
36:     for i in range(n):
37:         for j in range(n):
38:             if i == j:
39:                 continue
40:             try:
41:                 # Try enabling P2...

This code fragment shows two test loops. The first (lines 30-33) checks P2P access capability — it queries the CUDA driver's topology information to see whether the driver believes P2P is possible between each pair of GPUs. The second (lines 35-41) begins a P2P transfer test section, which would attempt actual data transfers between GPUs to verify that P2P works end-to-end, not just at the capability level.

The assistant is reading this code to understand what the existing script does and, crucially, what it does not do. The existing test checks can_device_access_peer (a software capability flag) but the assistant needs to verify actual data correctness — whether bytes written by one GPU can be correctly read by another. This distinction is vital because DmaRemapPeerMmio=1 might cause the driver to report P2P capability (the flag returns True) while the actual IOMMU mappings might be incomplete or faulty, leading to silent data corruption or IO_PAGE_FAULTs.

The Assumption Being Tested

The assistant is operating under a specific hypothesis: that the nvidia driver version 590.48.01, with DmaRemapPeerMmio=1 enabled by default, might already handle P2P DMA correctly under IOMMU translation on Blackwell GPUs. This is a reasonable assumption — the parameter exists precisely for this purpose, and the driver developers at NVIDIA would have had to account for systems with IOMMU enabled.

However, there are reasons to be skeptical. Earlier in the session (before the conversation segment captured here), the assistant had observed IO_PAGE_FAULTs during P2P operations. Those faults might have been caused by the identity-domain experiments corrupting GPU state, or they might indicate that DmaRemapPeerMmio=1 doesn't work correctly on this hardware/driver combination. The assistant needs to distinguish between these possibilities.

There's also an implicit assumption that torch.cuda.can_device_access_peer() returning True would correlate with actual working P2P transfers. This is not guaranteed — the CUDA runtime's topology detection might report capability based on PCIe topology alone, without accounting for IOMMU configuration. The actual test of P2P is whether a cross-GPU memcpy produces correct data.

Input Knowledge Required

To understand this message, the reader needs knowledge of several domains:

GPU architecture: Understanding that modern GPUs have a Firmware Security Processor (FSP/GSP) that handles initialization and requires specific DMA mappings from the kernel. Blackwell (SM120) is NVIDIA's latest GPU architecture, and its FSP behavior under IOMMU is apparently different from previous generations.

IOMMU concepts: Understanding DMA translation vs. identity (passthrough) modes, IOMMU groups, and how the Linux kernel's IOMMU subsystem manages DMA remapping for PCIe devices. The DMA-FQ and identity domain types, and the sysfs interface for switching between them.

Linux PCI subsystem: Knowledge of device removal, Secondary Bus Reset, driver binding/unbinding, and the rescan mechanism. The assistant had been manipulating /sys/bus/pci/devices/*/remove, /sys/bus/pci/drivers/nvidia/unbind, and /sys/bus/pci/rescan.

NVIDIA driver internals: Understanding that DmaRemapPeerMmio is a module parameter that controls IOMMU remapping for peer GPU BARs, and that it's distinct from the CUDA runtime's P2P capability detection.

PyTorch/CUDA programming: The torch.cuda.can_device_access_peer() function, which queries the CUDA driver's topology API to determine whether direct peer access is possible between two devices.

Output Knowledge Created

This message itself creates minimal output — it's a read operation that brings code into the assistant's context window. But it sets the stage for the following messages:

In [msg 6330], the assistant writes a new focused test script (p2p_test.py) that goes beyond the existing gpu_diag.py. The new script performs actual P2P transfers and validates data correctness, not just capability queries. This is the direct output of the knowledge gained from reading lines 30-41 — the assistant saw that the existing test was incomplete and needed to be supplemented.

In [msg 6332], the assistant runs the test and discovers that all P2P pairs report canAccessPeer=True under the current configuration (DMA-FQ mode with DmaRemapPeerMmio=1). This is a significant finding — it means the nvidia driver does report P2P capability even under IOMMU translation. The follow-up question, addressed in subsequent messages, is whether actual data transfers work correctly.

The Thinking Process

The assistant's reasoning in this message is implicit but clear. Having just spent many messages chasing a dead end (IOMMU identity domains), it has identified a new variable to investigate: DmaRemapPeerMmio. The logical next step is to test whether P2P works right now, in the current working configuration, before attempting any further modifications.

The decision to read the existing gpu_diag.py rather than immediately writing a new test shows methodical thinking. The assistant checks what infrastructure already exists, assesses its adequacy, and only then creates new code. This is the behavior of an experienced engineer who knows that the fastest path to an answer is often to use what's already available, modifying only what's necessary.

The specific lines requested (30-41) reveal what the assistant is focused on: the P2P capability check and the beginning of the transfer test. It's not interested in the single-GPU tests or the NCCL all-reduce test at this point — it wants to isolate the P2P question as cleanly as possible.

A Broader Perspective

This message, for all its brevity, captures a crucial moment in any complex debugging session: the pivot. The assistant had invested significant effort in one approach, hit a fundamental blocker (Blackwell FSP incompatibility with identity IOMMU), and is now reorienting toward a different strategy. The read tool call is the first step of that reorientation — gathering information before acting.

The article's subject message is a reminder that in technical work, the most important decisions are often invisible. The assistant could have continued down the IOMMU identity path, trying kernel boot parameters, udev rules, or other exotic configurations. Instead, it recognized that the nvidia driver itself might already contain the solution, hidden in a parameter it hadn't examined before. This recognition — that the problem might be solvable within the existing framework rather than requiring a fundamental reconfiguration — is the kind of insight that separates productive debugging from wheel-spinning.

The read tool call in [msg 6329] is not just reading code. It is reading the map of the territory, deciding where to dig next.