The Udev Insight: A Pivotal Shift in the Battle for GPU P2P DMA on Blackwell

In the ongoing struggle to enable peer-to-peer (P2P) DMA communication between NVIDIA Blackwell GPUs behind an IOMMU in full translation mode, message [msg 6305] marks a critical inflection point. After a long sequence of failed attempts — SBR (Secondary Bus Reset) that couldn't clear the GPU's Firmware Security Processor (FSP) state, fundamental resets that were unsupported, D3cold transitions that left GPUs in an unrecoverable "unknown" power state, and nvidia_gpu_tools.py recovery commands that returned silently without effect — the assistant arrives at a new hypothesis. The message is brief but conceptually dense: it contains a moment of genuine insight, a pivot in strategy, and the execution of a recovery operation to restore the system to a known-good baseline.

The Problem That Wouldn't Yield

To understand why this message matters, one must appreciate the dead end the assistant had reached. The goal was deceptively simple: enable P2P DMA between four NVIDIA RTX PRO 6000 Blackwell GPUs on a system where the AMD IOMMU was in full translation mode. The nvidia driver's DmaRemapPeerMmio=1 parameter was already enabled, but it produced incomplete IOMMU mappings — some peer GPU pairs worked while others caused IO_PAGE_FAULT errors. The alternative approach was to set individual IOMMU groups to "identity" mode, which bypasses DMA translation entirely for those devices, allowing direct GPU-to-GPU memory access.

The assistant had tried setting identity domains at every conceivable moment: while the GPUs were bound to nvidia (failed because the driver was actively using DMA mappings), after unloading nvidia and setting identity before reloading (the GPUs went into a broken state where nvidia-smi reported "No devices were found"), and after removing the PCI devices, performing SBR, and rescannning (the IOMMU groups were destroyed on removal and recreated with the default type on rescan). Each attempt ended with the same error pattern in dmesg: RmInitAdapter: Cannot initialize GSP firmware RM followed by 0xbadf4100 register reads — a signature of the Blackwell GPU's firmware being in an unrecoverable state after a software reset.

The Core Discovery

Message [msg 6305] opens with a critical empirical finding:

Group is destroyed. So we can't set identity while the device is removed. The group gets recreated on rescan with default type.

This is the crystallized lesson from the previous message ([msg 6304]), where the assistant had verified through a direct experiment that removing a GPU (echo 1 > /sys/bus/pci/devices/$gpu/remove) destroys its IOMMU group sysfs entry. The group directory at /sys/kernel/iommu_groups/$group simply vanishes. When the PCI bus is rescanned and the device reappears, a new group is created with the kernel's default IOMMU type — which on this AMD system with iommu=pt (pass-through) enabled at the kernel command line, defaults to DMA-FQ (DMA with Fair Queuing), not identity.

This is a fundamental constraint of the Linux IOMMU subsystem: IOMMU groups are tied to PCI device lifetime. They are not persistent across device removal and re-enumeration. There is no way to "pre-set" an IOMMU group type for a device that does not yet exist.

The Udev Hypothesis

The assistant then makes a conceptual leap:

Let me think about this problem differently. What if we use a udev rule that fires when the GPU appears and sets the IOMMU group to identity before nvidia probes? Udev rules run before driver probing.

This is the key insight of the message. The assistant realizes that the timing problem — needing to set identity before the nvidia driver initializes the GPU's FSP firmware — could be solved by hooking into the Linux device enumeration pipeline at the right moment. The udev device manager processes kernel uevents as devices are added and removed. When a PCI device appears on the bus, udev fires matching rules before the kernel's driver probe logic runs. If a udev rule could set the IOMMU group type to identity at this exact moment, the nvidia driver would find an identity-mode IOMMU group when it subsequently probes the device.

The assumption embedded in this hypothesis is that the IOMMU group type can be changed while the group exists but before any driver has bound to the device. The assistant had already demonstrated in [msg 6300] that setting identity on an unbound device works — the sysfs write succeeds — but the problem was that the nvidia driver subsequently failed to initialize. The udev approach would test whether the failure was due to the timing of the identity setting (set after nvidia had already partially initialized the GPU) or due to a fundamental incompatibility between identity mode and Blackwell's firmware.

Executing the Recovery

Before the udev rule can be implemented and tested, the system must be restored to a working state. The previous experiments had left the GPUs in a degraded condition. The assistant executes a recovery sequence:

  1. A bridge SBR (Secondary Bus Reset) on the PCIe upstream bridge 0000:00:01.1
  2. A 5-second wait for the reset to complete
  3. A PCI bus rescan (echo 1 > /sys/bus/pci/rescan)
  4. Another 5-second wait for driver probing to complete
  5. A verification via nvidia-smi The output confirms success: all four GPUs are detected and working:
0, 00000000:01:00.0
1, 00000000:11:00.0
2, 00000000:61:00.0
3, 00000000:71:00.0

This is important context for the reader: the SBR + rescan cycle, when performed without any identity-domain manipulation, reliably restores the GPUs to a functional state. The nvidia driver auto-probes the freshly enumerated devices, the FSP boots successfully, and all four GPUs appear in nvidia-smi. This confirms that the hardware itself is sound — the problem is specifically with the interaction between IOMMU identity mode and the Blackwell firmware initialization sequence.

Assumptions and Their Consequences

The assistant makes several assumptions in this message that deserve scrutiny:

Assumption 1: Udev rules run before driver probing. This is generally true for modern Linux systems. When the kernel discovers a new device, it emits a KOBJ_ADD uevent. Udev processes this event and runs matching rules. The kernel's driver probe typically occurs during the same device-add path, but udev rules can execute in the window between device discovery and driver binding. However, the precise ordering depends on kernel configuration, the udev daemon's scheduling, and whether the driver is built-in or a module. For PCI devices with modular drivers (like nvidia), the driver module must be loaded first, which adds another layer of timing complexity.

Assumption 2: The IOMMU group type can be changed atomically before any DMA mapping is created. The assistant assumes that writing "identity" to /sys/kernel/iommu_groups/$group/type at the right moment will take effect before the nvidia driver creates any DMA mappings. This is plausible but unverified — the IOMMU subsystem may have internal state that is set during group creation and not easily altered.

Assumption 3: The Blackwell FSP failure is a timing issue, not a fundamental incompatibility. This is the assumption that ultimately proves incorrect, as revealed in the subsequent chunk. The FSP (Firmware Security Processor) on Blackwell GPUs apparently requires specific DMA mappings that are set up by the kernel's DMA API in translation mode. Identity mode bypasses this DMA translation entirely, which means the FSP cannot establish the communication channels it needs during boot. This is not a matter of setting identity "too late" — it is a hardware/firmware requirement that cannot be satisfied without DMA translation.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several Linux kernel subsystems:

Output Knowledge Created

This message produces several important outputs:

  1. The confirmed constraint: IOMMU groups are destroyed and recreated with default type during PCI device removal and rescan, making pre-setting impossible
  2. A new strategy: The udev rule approach, which attempts to set identity in the window between device discovery and driver probing
  3. A restored working state: The four GPUs are back in service after the SBR + rescan recovery
  4. A testable hypothesis: That the Blackwell FSP failure is a timing issue solvable by earlier identity domain configuration

The Thinking Process

The reasoning visible in this message reveals a methodical, hypothesis-driven approach. The assistant begins by stating the newly learned constraint ("Group is destroyed"), then explicitly signals a cognitive shift ("Let me think about this problem differently"). This is followed by the articulation of a new hypothesis ("What if we use a udev rule...") and the reasoning supporting it ("Udev rules run before driver probing").

The structure mirrors the classic scientific method: observation (groups are destroyed), hypothesis (udev can set identity before driver probing), prediction (this will allow nvidia to initialize with identity mode), and experiment setup (restore GPUs to working state). The experiment itself — implementing and testing the udev rule — occurs in the subsequent messages, but the conceptual groundwork is laid here.

The message also demonstrates a key engineering skill: knowing when to stop hammering on a failing approach and pivot. The assistant had invested significant effort in the remove-SBR-rescan cycle, trying it with various modifications (blacklists, fundamental reset flags, gpu-admin-tools). Recognizing that the fundamental constraint was one of timing rather than mechanism, the assistant pivots to a completely different strategy rather than continuing to optimize the failing one.

Conclusion

Message [msg 6305] is a turning point in the P2P DMA restoration effort. It captures the moment when the assistant synthesizes the lessons from a series of failures into a new, more promising approach. The udev hypothesis is elegant in its simplicity: if the problem is that identity mode must be set before the GPU firmware initializes, then hooking into the device enumeration pipeline is the natural solution. While this particular approach ultimately fails (as revealed in the chunk summary — Blackwell's FSP fundamentally requires DMA translation mode), the reasoning process itself is instructive. It shows how deep system debugging requires not just technical knowledge but the ability to reframe problems, question assumptions, and design precise experiments to test hypotheses.