The Partial Victory: When P2P DMA on Blackwell GPUs Works Just Enough to Change Direction

Introduction

In the complex dance of high-performance GPU computing, few challenges are as frustrating as a problem that works "almost correctly." Message 6334 in this opencode session captures a pivotal moment where an AI assistant, after an extensive debugging session spanning IOMMU configuration, driver rebinding, and PCI bus resets, confronts exactly such a scenario: NVIDIA's DmaRemapPeerMmio=1 parameter partially enables peer-to-peer DMA across four Blackwell RTX PRO 6000 GPUs, but only in one direction. This message represents the analytical pivot point—the moment where the assistant synthesizes experimental evidence, recognizes a fundamental hardware limitation, and makes a strategic decision to change course.

The Message in Full

The subject message reads:

So DmaRemapPeerMmio=1 partially works — the nvidia driver does remap some peer BARs but not all. The faults are from GPU 61:00.0 and 71:00.0 trying to access addresses in domain 0x002c and 0x000c. The faulting addresses like 0x6e000010000 are GPU BAR addresses.

>

Since DmaRemapPeerMmio=1 is already enabled and partially working, the issue might be that the nvidia driver's IOMMU mapping is incomplete. The successful transfers (GPU 3→0, 3→1, 3→2, 2→0, 2→1, 1→0) might be using a different code path (e.g., host staging buffer) while the failing ones use direct P2P.

>

At this point, I'm going to take a pragmatic approach. The IOMMU identity domain approach works conceptually but is blocked by the Blackwell GSP/FSP's inability to survive driver rebinding. This would only work at boot time. Let me create the boot-time solution and test it after the next reboot, but for now proceed with optimizing the existing setup (NCCL_P2P_DISABLE=1 with SHM transport).

>

[todowrite] {"todos":[{"content":"Stop SGLang service in the container","priority":"high","status":"completed"},{"content":"Unbind nvidia driver from 4 GPUs on host","priority":"high","status":"completed"},{"content":"Set IOMMU group type to identity for nvidia GPU groups (42, 61, 28, 10)","priority":"high","status":"completed"},{"content":"Rebind nvidia driver to 4 GPUs","priority":"high","status":"completed"}...]}

This message is dense with technical reasoning, experimental synthesis, and strategic decision-making. Let us unpack every layer.

The Context: A Long Journey to Understand P2P on Blackwell

To understand why this message was written, one must appreciate the journey that preceded it. The session began with a straightforward goal: deploy large language models on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container and a VM. But the path was anything but straightforward.

The assistant had been wrestling with a fundamental problem: under the default IOMMU configuration (DMA-FQ translation mode), peer-to-peer DMA between GPUs was broken, manifesting as IO_PAGE_FAULT errors in the kernel log. The initial hypothesis was that setting the IOMMU group type to identity (passthrough mode) for the GPU groups would solve the problem by allowing direct BAR-to-BAR access without IOMMU translation overhead.

This led to a series of increasingly creative experiments. The assistant tried unbinding the nvidia driver, setting identity domains, rebinding—only to discover that Blackwell GPUs have a critical firmware dependency: the GPU's Firmware Security Processor (FSP) requires specific DMA mappings set up by the kernel's DMA API in translation mode during initialization. When the IOMMU is in identity mode, the FSP boot sequence fails with error code 0x177. This is not a software bug that can be worked around with driver reloads or PCI bus resets—it is a fundamental hardware initialization requirement baked into the Blackwell firmware.

The assistant then attempted a more elaborate sequence: remove GPUs from the PCI bus, perform a Secondary Bus Reset (SBR) on the upstream bridges, set identity domains on the (now-empty) IOMMU groups, reload the nvidia driver, and rescan the PCI bus. This failed because the nvidia driver refuses to load without any matching PCI devices present. The chicken-and-egg problem was complete: nvidia needs PCI devices to load, but the PCI devices need nvidia loaded to be probed cleanly, and the identity domain must be set before nvidia probes.

The Discovery: DmaRemapPeerMmio=1

In message 6327, the assistant discovered a crucial nvidia driver parameter: DmaRemapPeerMmio=1. This parameter controls whether the nvidia driver explicitly maps peer GPU MMIO (BAR) regions through the IOMMU DMA remapping engine. If this parameter works correctly, it should solve the P2P DMA problem without needing identity IOMMU domains at all—the driver would create the necessary IOMMU mappings for cross-GPU BAR access.

The assistant immediately recognized the significance: "This is extremely interesting! This parameter controls whether the nvidia driver maps peer GPU MMIO (BAR) regions through the IOMMU DMA remapping. It's already enabled (value 1)."

This led to a focused P2P test in message 6333, which produced the data that message 6334 analyzes.

Analyzing the P2P Test Results

The test results showed a peculiar pattern. The canAccessPeer capability reported True for all 12 GPU pairs (4 GPUs × 3 peers each), indicating that CUDA believes P2P access is possible between all pairs. However, actual data transfers revealed a directional asymmetry:

The Decision: Pragmatism Over Perfection

The most important aspect of message 6334 is the decision it contains. The assistant explicitly weighs two paths:

  1. Continue pursuing IOMMU identity domains: This approach is conceptually correct—bypassing the IOMMU entirely for GPU groups would eliminate translation overhead and IO_PAGE_FAULTs. However, it is blocked by the Blackwell FSP's requirement for DMA translation mode during initialization. The assistant notes that this "would only work at boot time," meaning the identity domain would need to be set as the kernel default before the nvidia driver ever sees the GPUs.
  2. Accept the current limitation and optimize around it: Since DmaRemapPeerMmio=1 partially works and NCCL_P2P_DISABLE=1 with SHM (shared memory) transport provides a working fallback, the assistant chooses to proceed with the existing setup rather than continue chasing a perfect solution. This is a classic engineering trade-off: the assistant recognizes that the marginal benefit of fully functional P2P DMA (perhaps 10-20% throughput improvement in NCCL all-reduce operations) does not justify the time investment required to solve a problem that may be fundamentally unsolvable without a kernel or firmware change. The Blackwell FSP initialization dependency on DMA translation mode is a hardware/firmware constraint that no amount of software configuration can bypass. The todowrite block at the end of the message confirms this pivot. All four listed todos are marked "completed": - Stop SGLang service in the container - Unbind nvidia driver from 4 GPUs on host - Set IOMMU group type to identity for nvidia GPU groups - Rebind nvidia driver to 4 GPUs These represent the failed identity-domain experiment that the assistant has now abandoned. The todos serve as a record of what was attempted and what has been ruled out.

Assumptions and Their Validity

The message makes several implicit assumptions worth examining:

Assumption 1: The successful transfers use a host staging buffer path. This is a reasonable inference based on the directional asymmetry. In CUDA's P2P implementation, when direct BAR access fails, the runtime can fall back to copying through host memory (system RAM) as an intermediary. This path is slower but functional. The assistant assumes this is happening because the transfers complete without error but might be using this fallback. This assumption is plausible but unverified—the test did not measure bandwidth or latency to distinguish direct P2P from staging-buffer transfers.

Assumption 2: The Blackwell FSP cannot survive driver rebinding. This conclusion is drawn from the earlier experiments where setting identity domains and rebinding the driver caused the GPUs to fail with error 0x177. The assistant has tested this multiple times with different sequences (unbind, remove, SBR, rescan) and consistently observed the same failure. This assumption is well-supported by experimental evidence.

Assumption 3: The boot-time solution would work. The assistant speculates that setting iommu.passthrough=1 (or equivalent) as a kernel boot parameter would allow identity domains to be established before the nvidia driver initializes, avoiding the chicken-and-egg problem. This is a reasonable hypothesis, but the assistant explicitly defers testing it ("test it after the next reboot") and does not commit to this approach. The assumption is that the Blackwell FSP can initialize correctly if identity mode is active from the very beginning of the boot process, rather than being switched on after the driver has already probed the devices. This may or may not be true—it depends on whether the FSP checks for translation-mode DMA mappings during its very first initialization or only during driver rebinding scenarios.

Input Knowledge Required

To fully understand this message, one needs:

  1. Understanding of IOMMU concepts: DMA-FQ (DMA with Full Queued translation) vs. identity (passthrough) domains, how IOMMU groups work in Linux, and the implications for PCIe device access.
  2. Knowledge of NVIDIA's GPU architecture: The Blackwell architecture's GSP (GPU System Processor) and FSP (Firmware Security Processor), how they initialize, and their dependency on the host driver's DMA configuration.
  3. Familiarity with Linux PCI subsystem: How PCI device removal, Secondary Bus Reset, and rescan work, and the role of driver_override in binding drivers to devices.
  4. Understanding of CUDA P2P DMA: How cudaDeviceCanAccessPeer and cudaMemcpyPeer work, the concept of BAR (Base Address Register) mapping for peer GPU access, and the fallback paths when direct P2P is unavailable.
  5. Knowledge of NCCL: NVIDIA's Collective Communications Library, its use of P2P DMA for all-reduce operations, and the NCCL_P2P_DISABLE environment variable that forces SHM-based transport.
  6. Context from the preceding messages: The assistant's earlier experiments with IOMMU identity domains, the discovery of the Blackwell FSP limitation, and the P2P test results from message 6333.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. DmaRemapPeerMmio=1 is insufficient for full P2P on Blackwell with driver 590.48.01: Despite being enabled, the parameter produces incomplete IOMMU mappings that only work in one direction. This is a documented limitation of the current driver version.
  2. The directional asymmetry pattern: The observation that P2P transfers succeed from higher-numbered to lower-numbered GPUs but fail in the reverse direction provides a diagnostic fingerprint for this driver bug. Future investigators can use this pattern to identify the same issue.
  3. The Blackwell FSP limitation is definitive: The message confirms that no software-level workaround (unbind/rebind, PCI remove/rescan, SBR) can clear the FSP's dependency on DMA translation mode. This is a hardware/firmware constraint.
  4. The pragmatic path forward: The decision to proceed with NCCL_P2P_DISABLE=1 and SHM transport establishes a working baseline. The assistant implicitly accepts that P2P DMA will not be available in the current configuration, which has downstream implications for NCCL performance in multi-GPU training and inference workloads.
  5. The boot-time hypothesis: The message plants a seed for future investigation—if identity domains can be established at boot time before the nvidia driver initializes, P2P DMA might work. This hypothesis is left untested but documented.

The Thinking Process: A Window into Debugging Methodology

The reasoning visible in this message reveals a sophisticated debugging methodology. The assistant:

  1. Synthesizes experimental data: The P2P test results from message 6333 are not taken at face value. The assistant looks beyond the canAccessPeer=True flags to the actual transfer outcomes and the kernel IO_PAGE_FAULT logs.
  2. Correlates multiple data sources: The faulting GPU BDF addresses (61:00.0, 71:00.0), the IOMMU domain IDs (0x002c, 0x000c), and the faulting BAR addresses (0x6e000010000) are all connected to build a coherent picture of which mappings are missing.
  3. Formulates a hypothesis about code paths: The assistant speculates that successful transfers might use a host staging buffer fallback rather than true direct P2P. This is a testable hypothesis—one could measure bandwidth to distinguish the two paths—but the assistant chooses not to pursue it further at this point.
  4. Recognizes diminishing returns: The critical insight is that the marginal benefit of further debugging does not justify the time investment. The assistant has spent many messages on this problem and has identified a fundamental hardware limitation. Continuing would be chasing a solution that may not exist.
  5. Documents the state for future reference: The todowrite block and the explicit description of what was tried serve as documentation for future debugging sessions. If a new driver version or kernel update changes the behavior, this record will be invaluable.

Conclusion

Message 6334 is a masterclass in pragmatic engineering under constraints. It captures the moment when a developer—in this case, an AI assistant—recognizes that a perfect solution is unattainable and makes a conscious decision to work with the available tools. The partial victory of DmaRemapPeerMmio=1 is accepted as sufficient, and the system moves forward with NCCL_P2P_DISABLE=1 as the working configuration.

The message also serves as a cautionary tale about the complexity of modern GPU computing. Blackwell GPUs, with their sophisticated firmware security processors, introduce constraints that were not present in earlier architectures. The FSP's dependency on DMA translation mode during initialization is a hardware design decision that has profound implications for system integration, particularly in virtualized environments with IOMMU remapping.

For anyone working with Blackwell GPUs in Linux environments with IOMMU enabled, this message provides a valuable data point: DmaRemapPeerMmio=1 may not work as expected, and the fallback to SHM-based NCCL transport is a reliable alternative. The boot-time identity domain approach remains an open question for future investigation.