The Modprobe Hook Gambit: A Turning Point in the Blackwell IOMMU Saga

Introduction

In the complex dance between GPU firmware, kernel memory management, and PCIe topology, timing is everything. Message [msg 6399] captures a pivotal moment in an extended troubleshooting session: the first test of a carefully crafted modprobe install hook designed to solve an intractable IOMMU identity domain problem on an 8-GPU NVIDIA Blackwell system. The message is deceptively simple — a single bash command dispatched to a remote host, executing a remove-SBR-rescan cycle with a new hook in place. But the outcome, "No devices were found," marks the death of one approach and the birth of a deeper understanding about Blackwell's Firmware Security Processor (FSP) and its relationship with the IOMMU.

This article examines that single message in depth: the reasoning that led to it, the assumptions it encoded, the knowledge it produced, and the thinking process visible in its construction.

Context: The Long Road to IOMMU Identity Domains

To understand message [msg 6399], one must first understand the problem it was trying to solve. The system in question was a Proxmox host with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs on NUMA0) and a SEV-SNP VM (4 GPUs on VFIO). The LXC's GPUs were behind an IOMMU in default DMA-FQ (DMA with Fine-grained Queuing) mode, which prevented direct peer-to-peer DMA between GPUs — a critical requirement for NCCL-based tensor parallelism in large language model inference.

The assistant had been pursuing a strategy of switching the IOMMU groups for the NUMA0 GPUs from DMA-FQ to identity mode, which would bypass IOMMU translation for those devices and enable P2P DMA. The challenge was timing: the NVIDIA driver module (nvidia) loads automatically when PCI devices are enumerated (triggered by the kernel's modalias mechanism), and it takes exclusive control of the GPUs during its probe callback. Once the driver is loaded, the IOMMU group type is effectively locked — the kernel refuses to change it while a driver is bound.

The preceding messages in the conversation ([msg 6384] through [msg 6398]) document a series of increasingly sophisticated attempts to solve this timing problem. The assistant tried unbinding the driver, removing the PCI devices, issuing Secondary Bus Reset (SBR) to reset the GPU state, then rescannning the PCI bus — all in an effort to create a window where the GPUs existed on the PCI bus but had no driver bound, allowing the IOMMU group type to be switched before NVIDIA's driver claimed them.

Each attempt revealed new constraints. The SBR approach worked once — in a previous session, after a fresh boot where the GPUs had never been touched by the NVIDIA driver. But subsequent attempts failed because the NVIDIA driver had already initialized the GPUs' GSP firmware, and SBR alone could not clear that state. The assistant discovered that modprobe nvidia would fail with "No such device" if no NVIDIA PCI devices were present on the bus, meaning the driver could not be pre-loaded before device enumeration. The modalias mechanism would then load the driver automatically during PCI rescan, binding the GPUs before any user-space script could intervene.

The solution seemed elegant: a modprobe install hook — a shell script registered in /etc/modprobe.d/ that would run instead of the real modprobe nvidia command. This script would first set the IOMMU identity domains for the target GPUs, then call the real modprobe nvidia with --ignore-install to load the actual driver. The hook was created in messages [msg 6393] through [msg 6398], and message [msg 6399] is its first test.

The Message: Anatomy of a Test

The message begins with a clear statement of intent:

Now let me test it. I'll do the remove+SBR+rescan cycle. This time when PCI rescan triggers modalias → modprobe nvidia, the install hook will run first, set identity domains, then load the real nvidia module:

The assistant then dispatches a single bash command to the remote host (root@10.1.2.6 — the Proxmox host). The command is a seven-step script:

  1. Unbind + rmmod: Unbind any existing NVIDIA driver instances and remove all NVIDIA kernel modules (nvidia_drm, nvidia_uvm, nvidia_modeset, nvidia).
  2. Remove GPUs: Write 1 to each GPU's /sys/bus/pci/devices/.../remove file to logically remove the devices from the PCI bus.
  3. SBR: Issue Secondary Bus Reset on each of the four PCI bridges upstream of the NUMA0 GPUs, by toggling the SBR bit in the bridge control register via setpci. A 15-second wait follows for the GSP firmware to reset.
  4. Block vfio-pci: Remove the NVIDIA PCI ID (10de 2bb5) from vfio-pci's driver override list, preventing vfio-pci from claiming the GPUs when they reappear.
  5. Rescan: Trigger a PCI bus rescan. This is the critical moment — the kernel will discover the GPUs, trigger modalias, which calls modprobe nvidia, which the install hook intercepts.
  6. Load supplementary modules: After the main NVIDIA module is loaded, load nvidia_modeset, nvidia_uvm, and nvidia_drm.
  7. Re-register vfio-pci: Re-add the NVIDIA PCI ID to vfio-pci so that the VFIO-bound GPUs (on NUMA1) remain bound to vfio-pci. The script concludes with diagnostic output: nvidia-smi to check GPU visibility, IOMMU group types for each GPU, and a journalctl query for log messages from the hook script (tagged with gpu-iommu). The output is devastatingly brief:
=== RESULT ===
No devices were found

No GPUs. No nvidia-smi output. No IOMMU type lines. No journalctl messages. The hook either didn't run, or ran but failed catastrophically.

The Reasoning Behind the Approach

The modprobe install hook approach was the culmination of a careful analysis of the kernel's PCI device enumeration sequence. The assistant had identified that the modalias mechanism works as follows:

  1. PCI rescan discovers new devices
  2. Kernel matches PCI vendor/device IDs against registered drivers
  3. For each unmatched device, kernel triggers modprobe via modalias
  4. modprobe loads the driver module
  5. Driver's probe callback binds the device The install hook intercepts step 4, inserting a pre-processing step (setting IOMMU identity) before the driver actually loads. The key insight was that between steps 3 and 4, the PCI device exists on the bus but has no driver bound — the IOMMU group should be mutable during this window. The assistant's reasoning was sound in principle. The modprobe install hook mechanism is a legitimate Linux kernel feature designed precisely for this kind of pre-processing. The hook script (/usr/local/bin/gpu-set-identity-before-nvidia.sh) was straightforward: iterate over the four GPU PCI addresses, check if the device exists and has no driver, set its IOMMU group to identity, then call the real modprobe nvidia with --ignore-install to bypass the hook recursively.

Assumptions and Their Consequences

The message encodes several assumptions, some explicit and some implicit:

Assumption 1: The modprobe hook would execute during PCI rescan. This was the core premise. The assistant assumed that when echo 1 > /sys/bus/pci/rescan triggered modalias, the kernel would call out to modprobe nvidia, which would be intercepted by the install hook. This assumption turned out to be correct — subsequent messages ([msg 6400]) confirmed the hook did run and successfully set identity domains. But the hook's success was irrelevant because of a deeper problem.

Assumption 2: SBR would clear the GSP firmware state. This was the critical error. The assistant had observed in a previous session that SBR followed by rescan worked to recover GPUs after driver unbind. But that success occurred on a system where the GPUs had never been initialized by the NVIDIA driver in the current boot session. In the current session, the GPUs had already been initialized (the driver had loaded at boot time, before the identity service could intervene), and the GSP firmware had entered a state that SBR could not reset. The error code 0x177 (visible in subsequent messages) indicated a GSP boot failure — the firmware was trying to use DMA mappings that only existed in translation mode, and identity mode broke them.

Assumption 3: The remove+SBR+rescan cycle would leave GPUs in a clean state. The assistant assumed that logically removing the PCI device, resetting the bridge, and rescannning would produce GPUs indistinguishable from a cold boot. This was incorrect for Blackwell GPUs, whose GSP firmware maintains persistent state across SBR. The GSP is a separate microcontroller within the GPU that handles power management, security, and firmware initialization. It appears to require specific DMA mappings set up by the kernel's DMA API during its boot sequence — mappings that only exist when the IOMMU is in translation mode.

Assumption 4: The hook script would be found and executable. The script was deployed via SCP in message [msg 6397] and made executable. But the modprobe hook runs in a minimal environment during early boot or module loading — could it find the script? Could it write to /sys/kernel/iommu_groups/.../type? These were implicit assumptions that turned out to be valid (the hook did run and succeeded at setting identity), but they were worth considering.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of several Linux kernel and PCIe concepts:

Output Knowledge Created

The message produced several important pieces of knowledge, even though the test "failed":

  1. The modprobe hook approach is technically viable but insufficient. The hook did run and set identity domains (confirmed in [msg 6400]). The failure was not in the hook mechanism but in the GSP firmware's incompatibility with identity mode.
  2. SBR does not reset Blackwell GSP firmware. This was the critical discovery. The GSP enters a state after NVIDIA driver initialization that persists across SBR. Once the driver has initialized the GPU, the GSP expects specific DMA mappings that only exist in translation mode.
  3. The Blackwell FSP boot sequence requires DMA translation. The error code 0x177 (GSP boot failure) and the "Possible bad register read" at address 0x110094 with value 0xbadf4100 indicate that the GSP firmware is trying to access memory through DMA mappings that don't exist in identity mode. This is a hardware/firmware limitation, not a software configuration issue.
  4. The timing window for identity mode is only at cold boot. The only way to get identity mode working is to have the modprobe hook in place before the NVIDIA driver ever touches the GPUs — meaning the hook must be active during the initial boot, when the GPUs have clean GSP firmware from power-on.
  5. Per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs. The chunk summary for segment 41 states this definitively: "Blackwell's FSP requires DMA translation mode during initialization, and no software-level reset (FLR, SBR, CXL bus reset) can clear this state."

The Thinking Process Visible in the Message

The assistant's thinking process is visible in several aspects of the message:

The structure of the test script reveals a careful, methodical approach. Each step is numbered and commented. Error handling uses || true to prevent the script from aborting on non-critical failures. The sleep durations (2s, 15s, 15s) show an understanding of PCIe timing requirements. The diagnostic output at the end (nvidia-smi, IOMMU types, journalctl) shows a desire for comprehensive feedback.

The choice of modprobe install hook over other approaches shows systems-level thinking. The assistant had tried direct approaches (unbind, set identity, rebind) and found that the NVIDIA driver's probe callback was too tightly coupled to device enumeration. The install hook is a clever workaround that exploits the kernel's own module loading infrastructure.

The language in the message reveals the assistant's mental model. The phrase "This time when PCI rescan triggers modalias → modprobe nvidia, the install hook will run first, set identity domains, then load the real nvidia module" shows a clear causal chain. The arrow notation (→) is particularly telling — it reveals how the assistant visualizes the sequence of events.

The absence of contingency handling is notable. The script has no error checking between steps 5 and 7. If the hook fails, the script continues blindly. This suggests the assistant was confident in the approach and didn't anticipate the GSP issue. The "No devices were found" output was clearly unexpected.

The Aftermath

The messages immediately following [msg 6399] ([msg 6400] through [msg 6404]) show the assistant's response to the failure. First, diagnostic commands reveal that the hook did work — identity domains were set — but the NVIDIA driver still failed with GSP errors. Then the assistant realizes the truth: the GPUs had already been initialized by the NVIDIA driver at boot time, before the modprobe hook could intervene. The SBR didn't clear the GSP state.

The assistant pivots: the modprobe hook is kept in place for the next cold boot, but for the current session, the GPUs are restored to working DMA-FQ mode. The systemd identity service is disabled as redundant. The system is returned to a stable state with SGLang serving Qwen3.5-122B-A10B BF16 on 4 GPUs with TP=4, MTP speculation enabled, and P2P DMA disabled via NCCL_P2P_DISABLE=1.

Conclusion

Message [msg 6399] is a turning point in a complex troubleshooting narrative. It represents the most sophisticated attempt to solve the IOMMU identity domain timing problem — a solution that was technically correct but defeated by an unforeseen hardware limitation. The message is a testament to the depth of systems knowledge required to manage modern GPU infrastructure, where kernel internals, PCIe topology, firmware behavior, and driver architecture all intersect.

The failure was not in the implementation but in the assumptions. The assistant assumed SBR would reset the GSP firmware, and that assumption was wrong. But the knowledge gained — that Blackwell's FSP requires DMA translation mode during initialization — is valuable and definitive. It closes one line of investigation and points toward others: the NVIDIA driver's DmaRemapPeerMmio=1 parameter, or accepting the DMA-FQ limitation and optimizing around it with MTP speculation.

In the end, the modprobe hook gambit failed, but it failed productively. It eliminated a plausible solution, revealed a fundamental hardware constraint, and redirected effort toward more promising optimizations. That is the nature of deep systems debugging: each failure narrows the search space, and the path forward is paved with dead ends.