The Moment P2P Was Almost There: Testing Blackwell Peer DMA Under IOMMU Translation

In the long and winding saga of getting eight NVIDIA RTX PRO 6000 Blackwell GPUs to communicate efficiently, message [msg 6332] represents a quiet but pivotal turning point. It is a single SSH command, executed against a remote server, running a freshly written Python test script. The output is deceptively clean: all twelve GPU-to-GPU peer access capability checks return True. Every GPU can see every other GPU. After hours of wrestling with IOMMU identity domains, Blackwell firmware boot failures, secondary bus resets, and nvidia driver rebinding gymnastics, this looks like a victory. But as the subsequent message reveals, the real story is more complicated — and more instructive.

The Context: A Long Road to Nowhere

To understand why this message matters, we must understand the dead end that preceded it. The assistant had been pursuing a strategy of setting IOMMU group types to identity for the four NUMA0 GPUs. In identity mode, the IOMMU performs no address translation for devices in that group — DMA addresses pass through directly to physical addresses. This is the ideal configuration for GPU P2P DMA, because peer-to-peer transfers between GPUs bypass the CPU entirely and operate over PCIe BAR (Base Address Register) ranges. With IOMMU translation active, those BAR addresses need to be mapped into the IOMMU page tables, and if the mappings are incomplete or incorrect, the GPU attempting the transfer triggers an IO_PAGE_FAULT and the transfer fails.

The identity domain approach had seemed promising. The assistant had written a modprobe install hook that would set the IOMMU groups to identity before the nvidia driver loaded, theoretically giving the driver a clean, translation-free environment to work with. But after a reboot to test this approach, a devastating discovery emerged: the Blackwell GPU's Firmware Security Processor (FSP) fails to boot with error code 0x177 when IOMMU is in identity mode. The FSP apparently requires specific DMA mappings that are set up by the kernel's DMA API during device initialization — mappings that only exist when the IOMMU is in translation mode. This is not a timing issue or a driver version problem; it is a fundamental hardware requirement. Blackwell GPUs cannot initialize under identity IOMMU, period.

This discovery, documented in the chunk summary for segment 41, closed off the most promising path to P2P DMA restoration. The assistant had to pivot.

The Pivot: Can the Driver Fix What the IOMMU Cannot?

The alternative path was hiding in plain sight. Earlier, while examining nvidia driver parameters via /proc/driver/nvidia/params, the assistant had spotted a parameter called DmaRemapPeerMmio — already set to 1. This parameter controls whether the nvidia driver itself creates IOMMU DMA mappings for peer GPU MMIO (BAR) regions. If the driver handles the mapping correctly, P2P DMA should work even under full IOMMU translation mode (DMA-FQ), because the driver would ensure that each GPU's BAR range is properly registered in the IOMMU page tables for access by other GPUs.

The question was whether this mechanism actually worked on Blackwell with the 590.48 driver. The assistant had been running with NCCL_P2P_DISABLE=1 throughout the deployment, which sidesteps the issue entirely by forcing NCCL to use shared memory or other transports instead of direct P2P. But this carries a performance cost. If DmaRemapPeerMmio=1 could make P2P work correctly, the assistant could remove the NCCL override and potentially see significant throughput improvements.

Message [msg 6332] is the test that answers this question.

The Message Itself: A Clean Test with Hidden Depths

The command is straightforward:

ssh root@10.1.230.174 'NCCL_P2P_DISABLE="" ~/ml-env/bin/python3 /root/p2p_test.py 2>&1'

The key detail is NCCL_P2P_DISABLE="" — explicitly setting the environment variable to an empty string, overriding any previous setting. This ensures that NCCL (and by extension, the CUDA runtime) will attempt to use P2P DMA if it is available. The test script p2p_test.py was written by the assistant in the previous message ([msg 6330]) and copied to the remote machine via scp ([msg 6331]). It is a focused diagnostic tool designed to test two things: whether CUDA reports that peer access is possible (can_device_access_peer), and whether actual data transfers between GPU pairs succeed and produce correct results.

The output begins:

GPUs: 4
NCCL_P2P_DISABLE: not set

=== P2P Access Capability ===
  GPU 0 -> GPU 1: canAccessPeer=True
  GPU 0 -> GPU 2: canAccessPeer=True
  GPU 0 -> GPU 3: canAccessPeer=True
  ...

All twelve directional pairs report canAccessPeer=True. This is the CUDA runtime's opinion about whether the hardware topology supports direct peer access — it checks PCIe topology, BAR sizes, and whether the devices are on the same root complex or have appropriate PCIe switches. On this system, with all four GPUs connected through PCIe switches to the same EPYC socket, the answer is uniformly yes.

The message is truncated in the conversation data, but the full output (visible in the subsequent message [msg 6333]) reveals the critical finding: despite all canAccessPeer checks returning True, the actual P2P transfer tests reveal a directional failure pattern. Transfers from GPU 0 to any other GPU fail. Transfers from GPU 1 to GPUs 2 and 3 fail. But transfers from higher-numbered GPUs to lower-numbered ones succeed. This is not random — it suggests a systematic issue with how the nvidia driver's DmaRemapPeerMmio mechanism creates IOMMU mappings.

The Thinking Process: Reading Between the Lines

The assistant's reasoning in this moment is visible through the sequence of actions. Having been blocked on the identity domain approach, the assistant is now testing a hypothesis: "Maybe DmaRemapPeerMmio=1 already solves the problem, and we don't need identity domains at all." This is an optimistic hypothesis — it would mean the existing configuration (DMA-FQ IOMMU, DmaRemapPeerMmio=1) is already correct and the NCCL_P2P_DISABLE workaround can be removed.

The test is designed to be definitive. By explicitly clearing NCCL_P2P_DISABLE, the assistant ensures no environment-level override interferes. By writing a custom test script rather than relying on existing diagnostics, the assistant controls exactly what is measured — both the capability check and the actual data transfer. The script presumably allocates tensors on each GPU, attempts to copy between them using torch.distributed or raw CUDA, and verifies data integrity.

The assumption embedded in this test is that canAccessPeer=True is a reliable predictor of functional P2P DMA. This assumption turns out to be incorrect — or rather, incomplete. CUDA's can_device_access_peer checks topology and hardware capability, but it does not verify that the IOMMU mappings are actually in place. The nvidia driver's DmaRemapPeerMmio mechanism may report success for the mapping setup but leave some BAR ranges unmapped, or may create mappings that work in one direction but not the other due to how the IOMMU page tables are configured.

Input Knowledge Required

To fully understand this message, one needs:

  1. CUDA P2P semantics: The distinction between can_device_access_peer (a capability query) and actual P2P transfers (which require working IOMMU mappings or identity domains).
  2. IOMMU modes: DMA-FQ (translation with flush queues) vs identity (passthrough). The system is currently in DMA-FQ mode after the identity domain approach failed.
  3. Nvidia driver internals: The DmaRemapPeerMmio parameter, which tells the nvidia driver to create IOMMU mappings for peer GPU BARs. This is a relatively obscure parameter, not commonly discussed in public documentation.
  4. Blackwell architecture specifics: The FSP boot requirement for DMA translation mode, which makes identity domains impossible regardless of timing or driver reloading strategy.
  5. The PCIe topology: Four GPUs on NUMA0, connected through PCIe bridges (00:01.1, 10:01.1, 60:01.1, 70:01.1), with the system running on an AMD EPYC processor with AMD-Vi IOMMU.
  6. The session history: The assistant has been debugging P2P issues for hours, trying unbind/rebind cycles, SBR sequences, modprobe hooks, and various IOMMU configurations.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. P2P capability is confirmed: All four GPUs report peer access capability, confirming the PCIe topology supports direct GPU-to-GPU communication.
  2. DmaRemapPeerMmio is active but incomplete: The subsequent message ([msg 6333]) reveals that actual transfers fail in a directional pattern, indicating the nvidia driver's IOMMU remapping is not fully functional.
  3. The NCCL_P2P_DISABLE workaround remains necessary: Since P2P transfers are unreliable even with DmaRemapPeerMmio=1, the assistant must keep NCCL_P2P_DISABLE=1 and rely on alternative communication paths.
  4. A new debugging direction: The directional failure pattern provides a clue — the IOMMU mappings are asymmetric. This could be due to the order in which GPUs register their BARs, or a limitation in how many peer mappings the driver creates.

The Broader Significance

Message [msg 6332] sits at a critical juncture in the session. It represents the moment when the assistant exhausts the IOMMU configuration approach and pivots to accepting the nvidia driver's built-in mechanisms — only to find them partially broken. The subsequent decision, documented in [msg 6334], is a pragmatic one: stop chasing P2P DMA and optimize the existing NCCL_P2P_DISABLE configuration. The assistant updates its todo list, marking the identity domain experiments as completed (and failed), and shifts focus to MTP speculation and other optimizations that do not require P2P.

This is a pattern familiar to anyone who has debugged GPU clusters at the hardware level: the clean solution (identity IOMMU) is blocked by firmware requirements; the fallback solution (driver-managed remapping) is partially broken; the pragmatic solution (disable P2P and optimize around it) becomes the winner. The assistant's willingness to test hypotheses quickly, interpret failure patterns, and pivot without over-investing in any single approach is what makes this session effective.

The message also illustrates a subtle but important point about GPU diagnostics: can_device_access_peer is necessary but not sufficient for P2P DMA. It checks topology but not IOMMU configuration. A system can report peer access capability and still fail on actual transfers if the IOMMU page tables are incomplete. This is a trap that has caught many engineers, and the assistant's custom test script — which tests both capability and actual transfers — is exactly the right tool to avoid it.