The Modprobe Install Hook: A Last-Ditch Gambit for Blackwell GPU P2P DMA

In the high-stakes world of Blackwell GPU deployment, where every millisecond of inference throughput counts, the ability to enable Peer-to-Peer (P2P) Direct Memory Access between GPUs can make or break a production system. Message 6393 in this opencode session captures a pivotal moment: the assistant, after a long and frustrating sequence of failed attempts, pivots to a creative but desperate workaround — a modprobe install hook designed to set IOMMU identity domains before the NVIDIA driver can seize control of freshly enumerated Blackwell GPUs.

The Context: A War of Timing Against the Kernel

To understand why this message matters, one must appreciate the battlefield. The system in question runs Ubuntu 24.04 with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split across two NUMA domains. Four GPUs are bound to the host's NVIDIA driver for serving the Qwen3.5-122B-A10B model via SGLang; the other four are assigned to a confidential VM via VFIO. The host-side GPUs need P2P DMA to maximize multi-GPU communication for tensor parallelism, but a fundamental obstacle stands in the way: the IOMMU.

The IOMMU (Input/Output Memory Management Unit) sits between the GPUs and system memory, translating device DMA addresses to physical addresses. By default, it operates in "DMA-FQ" (DMA with Fine-Grained translations) mode, which provides full isolation but incurs overhead for every P2P transfer. The alternative is "identity" mode, where the IOMMU maps device addresses 1:1 to physical addresses, eliminating translation overhead and enabling direct P2P DMA. The problem is that identity mode must be set per IOMMU group before any driver binds to the device — and the NVIDIA driver, once loaded, takes exclusive ownership immediately upon PCI probe, locking the IOMMU type in place.

The previous messages in this segment (6390–6392) document a painful series of attempts to break this timing deadlock. The assistant tried removing GPUs from the PCI bus, performing Secondary Bus Resets (SBR) to clear GPU firmware state, blocking NVIDIA's modalias-triggered autoload, setting identity domains on unbound devices, and then loading NVIDIA — only to discover that the Blackwell GPU's GSP (GPU System Processor) firmware survives SBR and fails to initialize when the IOMMU is in identity mode, producing the dreaded error 0x177 and RmInitAdapter failed. Even when identity was set before NVIDIA loaded, the driver's probe callback would crash because the GSP firmware requires specific DMA mappings that only exist in translation mode.

The Message: A Creative Pivot to Modprobe Hooks

Message 6393 begins with a moment of recognition. After the previous attempt (message 6392) — where modprobe nvidia failed with "No such device" because the module couldn't load without any matching PCI devices present, but the subsequent PCI rescan triggered modalias-based loading which did work — the assistant articulates the core insight:

GPUs work again but with DMA-FQ. The modprobe nvidia failed ("No such device" — it needs at least one matching PCI device), but the rescan triggered modalias-based loading which worked.

>

So the pattern is: nvidia can only be loaded when PCI devices exist, and it takes exclusive control immediately during probe. There's no window to set identity between device enumeration and nvidia binding.

This is the crystallizing realization. The assistant has exhausted all conventional approaches: unbind-and-rmmod cycles, SBR, manual driver probing, blocking modalias autoload. Every path leads to the same dead end — NVIDIA's driver binding is atomic from the perspective of userspace manipulation. There is no race window to exploit.

The response is a lateral move: instead of trying to insert a timing gap after device enumeration but before driver binding, the assistant decides to intercept the driver loading process itself using a Linux modprobe install hook:

The only remaining approach is to use a modprobe install hook that sets identity domains before the nvidia module actually loads:

The command creates a configuration file at /etc/modprobe.d/nvidia-iommu-identity.conf:

# Before nvidia module loads, set IOMMU identity domains for NUMA0 GPUs
# This runs when modprobe is triggered (e.g., by PCI rescan modalias)
install nvidia /usr/local/bin/gpu-set-identity-before-nvidia.sh

How the Modprobe Install Hook Mechanism Works

The install directive in modprobe configuration is a powerful and dangerous tool. Normally, when modprobe nvidia is invoked (whether manually or via udev's modalias trigger), the module loading infrastructure locates the nvidia.ko kernel object, resolves dependencies, and inserts it into the running kernel. The install directive replaces this default behavior with a custom command — in this case, a shell script that presumably:

  1. Identifies the IOMMU groups corresponding to the NUMA0 Blackwell GPUs (based on their PCI addresses 0000:01:00.0, 0000:11:00.0, 0000:61:00.0, 0000:71:00.0).
  2. Writes "identity" to each group's /sys/kernel/iommu_groups/$group/type file.
  3. Then proceeds to actually load the nvidia module, likely by calling modprobe --ignore-install nvidia to bypass the hook recursively. The critical assumption is that the IOMMU group type can be changed while the group contains PCI devices that are present on the bus but not yet bound to any driver. This is a narrow window: after PCI rescan enumerates the devices but before the modalias trigger causes modprobe nvidia to run. The hook intercepts that modprobe invocation, giving the script a chance to set identity domains before the NVIDIA driver's probe callback executes.

Assumptions Embedded in This Approach

This approach makes several assumptions, some of which are quite fragile:

First, it assumes that the IOMMU group type is mutable while devices are present but unbound. The kernel's IOMMU subsystem may reject writes to the type file if any device in the group is already bound to a driver, but the assistant is betting that unbound devices are acceptable. This is not guaranteed — the IOMMU core code may require the group to be completely empty of devices before allowing a type change.

Second, it assumes that the Blackwell GSP firmware will initialize correctly under identity mode when the NVIDIA driver probes after identity has been set. This directly contradicts the evidence from message 6391, where identity mode caused GSP boot failures with error 0x177. The assistant seems to be hoping that the timing matters — that if identity is set before NVIDIA's probe rather than after, the driver's initialization path might handle it differently. This is a thin hope.

Third, it assumes that the modprobe hook executes synchronously and blocks the modalias-triggered loading until the script completes. If udev or the kernel's async firmware loading path spawns the modprobe call in a way that doesn't wait for the install hook to finish, the race condition could re-emerge.

Fourth, it assumes that the script path /usr/local/bin/gpu-set-identity-before-nvidia.sh exists and is executable at the time the hook fires. The message only creates the modprobe configuration file; the script itself is not shown being created, suggesting it either already exists from a previous step or will be created in a subsequent message.

What Went Wrong: The Fatal Flaw

The message's reasoning is sound in its analysis of the timing problem, but it carries a fatal flaw that becomes apparent only in the next segment (chunk 0 of segment 41). The Blackwell GPU's Firmware Security Processor (FSP) — the component that initializes the GPU's internal state during driver probe — requires DMA translation mode to be active during its boot sequence. Identity mode breaks the FSP initialization because the FSP relies on specific DMA mappings that the kernel's DMA API sets up in translation mode. No amount of timing manipulation can fix this: the FSP will fail with error 0x177 regardless of whether identity is set before or after the driver loads.

The assistant's earlier experiments (messages 6390–6392) had already demonstrated this: setting identity domains on unbound GPUs and then triggering NVIDIA's driver probe resulted in RmInitAdapter failed and Cannot initialize GSP firmware RM. The modprobe hook approach does not change the fundamental hardware requirement — it only changes the sequence of software events. The Blackwell FSP's need for translation-mode DMA mappings is a hardware-level constraint that cannot be circumvented by software timing tricks.

Input Knowledge Required

To fully appreciate this message, the reader needs familiarity with several Linux kernel concepts: the PCI subsystem (device enumeration, bus rescan, driver binding), the IOMMU subsystem (DMA-FQ vs. identity domains, per-group type control), the modprobe and module loading infrastructure (modalias triggers, install hooks), and NVIDIA's proprietary driver architecture (GSP firmware, RM initialization). Additionally, understanding the Blackwell GPU architecture — particularly the FSP's role and its sensitivity to IOMMU configuration — is essential to grasp why this approach ultimately fails.

Output Knowledge Created

This message creates several pieces of knowledge for the session:

  1. The timing deadlock is formally identified: NVIDIA's driver binds atomically upon PCI probe, leaving no userspace window to set IOMMU identity domains. This is a precise characterization of the problem.
  2. The modprobe install hook is proposed as the only remaining approach: This is a creative application of a little-used Linux module loading feature to solve a race condition that conventional methods cannot address.
  3. The approach's assumptions are implicitly documented: By choosing this path, the assistant implicitly acknowledges that all other approaches (SBR, manual unbind/bind, modalias blocking) have been exhausted.
  4. The script's purpose is defined: Even though the script body is not shown, the hook configuration specifies its location and intent, serving as a specification for what must be implemented.

The Thinking Process Revealed

The message reveals a clear chain of reasoning. The assistant starts by summarizing the outcome of the previous attempt ("GPUs work again but with DMA-FQ"), then analyzes why the attempt partially succeeded (modalias-triggered loading worked even though explicit modprobe failed). The key insight is stated with emphasis: "nvidia can only be loaded when PCI devices exist, and it takes exclusive control immediately during probe." The conclusion follows logically: "There's no window to set identity between device enumeration and nvidia binding."

The phrase "The only remaining approach" signals that this is a terminal decision point — the assistant has exhausted the search space of conventional solutions and is now reaching for an unconventional mechanism. The modprobe install hook is not a standard tool for IOMMU configuration; it's a creative hack that repurposes a module loading feature for a completely unrelated purpose.

Conclusion

Message 6393 represents a moment of creative desperation in a long debugging session. The assistant correctly identifies the fundamental timing constraint — NVIDIA's driver binding is atomic and immediate upon PCI enumeration — and proposes an ingenious workaround using modprobe's install hook mechanism. However, the approach is built on assumptions that directly contradict earlier experimental evidence about the Blackwell FSP's requirement for translation-mode DMA. The hook is ultimately a dead end, as confirmed in the subsequent segment where the FSP boot failure under identity mode is recognized as a hardware-level incompatibility that no software timing trick can overcome.

Yet the message is valuable precisely because it documents this dead end. It shows the assistant's systematic exploration of the solution space, its willingness to think laterally when conventional approaches fail, and its ability to precisely articulate the core constraint. In the broader narrative of the session, this message marks the point where the P2P DMA restoration effort is definitively abandoned, and the focus shifts to optimizing throughput through other means — namely, the MTP speculation that delivers 12–45% improvement without requiring P2P at all.