The Blackwell GSP Barrier: How a Modprobe Hook Became the Only Path to GPU P2P DMA

Introduction

In the high-stakes world of multi-GPU machine learning deployments, few things are as frustrating as discovering that a fundamental hardware capability—Peer-to-Peer (P2P) DMA between GPUs—is blocked not by a configuration error, but by the very firmware that makes the GPUs work. Message <msg id=6409> captures the moment when an AI assistant, after hours of painstaking debugging across multiple reboot cycles, arrives at a definitive understanding of a deeply entrenched hardware limitation. The message is a summary—a retrospective analysis that crystallizes everything learned during the session about the interaction between NVIDIA's Blackwell GPU architecture, IOMMU identity domains, and the GPU's Firmware Security Processor (FSP). It is not a command, not a tool call, not an action—it is a moment of clarity, a diagnosis delivered after the experiments have failed and the root cause has been identified.

This article examines that single message in detail: what it reveals about the Blackwell GPU architecture, the reasoning process that led to its conclusions, the assumptions that were tested and discarded, and the knowledge it creates for future work. The message is a turning point—the end of one line of inquiry and the beginning of another, where the assistant transitions from trying to fix a runtime problem to ensuring the fix is in place before the system even boots.

The Context: A Multi-Session Saga of GPU Configuration

To understand message <msg id=6409>, one must understand the broader context. The session is part of a larger effort to deploy large language models (specifically Qwen3.5-122B-A10B BF16) across multiple NVIDIA RTX PRO 6000 Blackwell GPUs on a Proxmox-hosted Ubuntu 24.04 system. The system has 8 Blackwell GPUs total, split between an LXC container and a VM, with 4 GPUs on NUMA node 0 and 4 on NUMA node 1. The NUMA0 GPUs are bound to the NVIDIA driver for SGLang serving, while the NUMA1 GPUs are bound to vfio-pci for use by a separate VM.

The specific problem that message <msg id=6409> addresses is the restoration of P2P DMA between the NUMA0 GPUs. Earlier in the conversation, the assistant discovered that P2P DMA was failing because the IOMMU was in translation mode (specifically DMA-FQ), which intercepts and remaps DMA addresses. For P2P DMA to work between GPUs on the same NUMA node, the IOMMU groups for those GPUs need to be set to identity mode, which bypasses address translation and allows direct GPU-to-GPU memory access.

The assistant had attempted to set these identity domains at runtime using a systemd service (gpu-iommu-identity.service) that would run at boot. But a boot-ordering problem caused the identity service to run after the NVIDIA driver had already claimed the GPUs, and a subsequent attempt to unbind and rebind the GPUs revealed a devastating property of Blackwell GPUs: once the NVIDIA driver's initial probe locks the GPU's Firmware Security Processor (FSP), no amount of software reset—not Function Level Reset (FLR), not Secondary Bus Reset (SBR), not even CXL bus reset—can clear it. The GPU becomes permanently corrupted until a full power cycle.

What the Message Says: A Summary of Hard-Won Knowledge

Message <msg id=6409> is structured as a retrospective summary. It begins with the problem statement: the boot-ordering issue where gpu-iommu-identity.service ran before gpu-vfio-split.service, causing the VFIO split script to grab all GPUs including the NUMA0 ones. When the assistant tried to fix this at runtime by unbinding and rebinding the NVIDIA driver, the Blackwell GPU's GSP firmware became permanently corrupted.

The root cause is stated with unusual clarity and certainty:

"On Blackwell GPUs, the nvidia driver's initial probe locks the GPU's Firmware Security Processor (FSP/GSP). Any subsequent driver unbind+rebind fails with kfspSendBootCommands_HAL error. The ONLY way to get working GPUs with identity IOMMU domains is to set the domains before nvidia ever touches the GPUs — at the very first module load."

This is the core insight of the entire session. The assistant has identified that the Blackwell FSP is fundamentally different from previous GPU architectures. On earlier NVIDIA GPUs (like Hopper/Ampere), unbinding and rebinding the driver was a routine operation—the GPU would simply reinitialize. On Blackwell, the FSP enters a locked state after the first driver probe, and no software-level reset can restore it. The only recovery is a full power cycle (cold boot) of the machine.

The solution is a modprobe install hook—a mechanism that intercepts the modprobe nvidia command before the module actually loads, sets the IOMMU identity domains for the NUMA0 GPUs, and then loads the real NVIDIA module. The hook is implemented as a shell script at /usr/local/bin/gpu-set-identity-before-nvidia.sh and triggered by a configuration file at /etc/modprobe.d/nvidia-iommu-identity.conf:

install nvidia /usr/local/bin/gpu-set-identity-before-nvidia.sh

This is a clever use of the install directive in modprobe.d, which allows a custom command to replace the normal module loading process. The script runs before the NVIDIA module is loaded, sets the IOMMU groups to identity, and then calls exec /sbin/modprobe --ignore-install nvidia to load the real module.

The Reasoning Process: A Trail of Failed Experiments

The thinking visible in the surrounding messages reveals a systematic debugging process. Let me trace the reasoning arc that leads to message <msg id=6409>.

Earlier in the session (messages <msg id=6390> through <msg id=6393>), the assistant attempted to set identity domains at runtime. It tried unbinding the NVIDIA driver, setting the IOMMU group type to identity, and then rebinding. The result was catastrophic: NVRM: GPU 0000:11:00.0: rm_init_adapter failed, device minor number 3 and the infamous gpuHandleSanityCheckRegReadError_GH100: Possible bad register read error. The GPUs were dead.

The assistant then tried a more elaborate approach: removing the PCI devices from the bus, performing a Secondary Bus Reset (SBR) on the upstream bridges, and then rescanning. This worked once before in a previous session, but it failed here. The key difference was that in the previous successful attempt, the NVIDIA driver had never been loaded during that boot session—the GPUs were fresh from power-on. In the current boot, the driver had already probed the GPUs, locking the FSP.

This led to the insight captured in message <msg id=6392>:

"The difference from the successful attempt is that nvidia was loaded in memory before the rescan — the nvidia driver's PCI probe callback handles the fresh-from-SBR device differently than a cold modalias-triggered module load + probe."

But even this turned out to be incorrect. The SBR doesn't clear the GSP at all. The assistant's earlier success was purely because the GPUs had never been touched by the NVIDIA driver in that boot session. The SBR was a red herring—it worked only because the GPUs were already clean.

Message <msg id=6393> shows the assistant pivoting to the modprobe install hook approach, recognizing that there's no window to set identity domains between PCI enumeration and NVIDIA driver binding:

"So the pattern is: nvidia can only be loaded when PCI devices exist, and it takes exclusive control immediately during probe. There's no window to set identity between device enumeration and nvidia binding."

The modprobe hook was created in messages <msg id=6393> through <msg id=6398>, with a bash heredoc parsing error (the $@ parentheses) requiring a workaround using a file write instead.

The test of the modprobe hook (message <msg id=6399>) showed that the identity domains were set correctly—the hook logged the changes to syslog—but the NVIDIA driver still failed because the GSP was already corrupted from the earlier bind in the same boot session. This confirmed both that the hook works and that the GSP corruption is permanent within a boot session.

Assumptions Made and Corrected

Several assumptions were tested and found incorrect during this session:

Assumption 1: SBR clears the Blackwell GSP. The assistant initially believed that a Secondary Bus Reset would reset the GPU's firmware state, based on a previous successful recovery. The correction was that the previous success was coincidental—the GPUs were clean because the NVIDIA driver had never loaded in that boot, not because the SBR cleared anything.

Assumption 2: There is a race window between PCI enumeration and driver binding. The assistant tried to set identity domains after PCI devices appeared but before the NVIDIA driver claimed them. The correction was that the NVIDIA driver loads via modalias triggered by PCI enumeration itself—there is no gap.

Assumption 3: The modprobe install hook would work mid-session. The assistant tested the hook during the same boot session where the GPUs had already been corrupted. The hook correctly set identity domains, but the NVIDIA driver still failed because the GSP was already locked. The correction was that the hook must be in place before the first NVIDIA module load at boot.

Assumption 4: Systemd service ordering would be sufficient. The assistant initially used a systemd service with After= dependencies to set identity domains before the NVIDIA driver loaded. This failed because the NVIDIA driver loads via modalias (triggered by udev), not via systemd, and the modprobe path doesn't respect systemd ordering.

Input Knowledge Required

To fully understand message <msg id=6409>, the reader needs knowledge of:

  1. IOMMU concepts: What DMA remapping is, what identity vs. translation mode means, and how IOMMU groups map to PCI devices.
  2. Linux kernel PCI subsystem: How PCI device enumeration works, how drivers bind to devices via modalias, and how modprobe.d install hooks function.
  3. NVIDIA GPU architecture: The difference between GSP (GPU System Processor) on older architectures and FSP (Firmware Security Processor) on Blackwell, and how the driver initialization sequence locks firmware state.
  4. SGLang deployment: The context of serving large language models across multiple GPUs with tensor parallelism, and why P2P DMA matters for performance.
  5. Proxmox/VFIO: How GPU passthrough works with vfio-pci, and how the system's 8 GPUs are split between host and VM.
  6. PCIe reset mechanisms: FLR, SBR, and CXL bus reset, and their effects on GPU state.

Output Knowledge Created

Message <msg id=6409> creates several important pieces of knowledge:

  1. Blackwell FSP locking is permanent within a boot session. This is a critical hardware behavior that anyone deploying Blackwell GPUs needs to know. Unbinding and rebinding the NVIDIA driver is not a safe operation on Blackwell—it corrupts the GPU until the next cold boot.
  2. The modprobe install hook pattern for pre-driver IOMMU configuration. This is a reusable technique: by using install nvidia /path/to/script in modprobe.d, you can execute arbitrary setup before the NVIDIA module loads, including IOMMU configuration, module parameter setting, or device preparation.
  3. The exact boot sequence required for P2P DMA on Blackwell with IOMMU. The message specifies a 6-step sequence that must be followed precisely: kernel boot, VFIO split, udev modalias trigger, modprobe hook (sets identity), NVIDIA module load (with identity already set), and GPU probe (clean GSP, success).
  4. Documentation of the error signature. The kfspSendBootCommands_HAL error and the 0xbadf4100 register read pattern are documented as diagnostic signatures for the locked-FSP condition.
  5. The limitation of software resets. The message explicitly states that FLR, SBR, and CXL bus reset all fail to clear the Blackwell FSP state. This is a definitive finding that saves future debuggers from trying these approaches.

Mistakes and Their Resolution

The most significant mistake visible in the session is the initial belief that SBR could clear the Blackwell GSP. This was a reasonable assumption based on previous experience with other GPU architectures and a coincidental success. The assistant corrected this by carefully comparing the conditions of the successful and failed attempts, identifying the critical variable: whether the NVIDIA driver had ever loaded during the boot.

A second mistake was the reliance on systemd service ordering. The assistant created gpu-iommu-identity.service with After=gpu-vfio-split.service dependencies, but the NVIDIA driver loads via udev/modalias, which is independent of systemd service ordering. The modprobe hook approach is the correct solution because it intercepts the module loading at the kernel level, before any driver code executes.

A third mistake was testing the modprobe hook in the same boot session where the GPUs were already corrupted. While this was a useful test—it confirmed the hook's mechanism worked—it couldn't validate the end-to-end success because the GSP was already locked. The assistant recognized this and correctly deferred full validation to the next reboot.

The Broader Implications

Message <msg id=6409> has implications beyond this specific deployment. The discovery that Blackwell GPUs cannot survive a driver unbind/rebind cycle affects any system that needs to reconfigure GPU bindings at runtime. This includes:

Conclusion

Message <msg id=6409> is a document of hard-won understanding. It represents the moment when a complex, multi-variable problem finally resolves into a clear picture. The assistant has traced the failure through multiple layers—from boot ordering to PCI bus resets to GPU firmware initialization—and arrived at a definitive root cause. The solution is elegant: a modprobe install hook that sets IOMMU identity domains before the NVIDIA driver ever touches the GPUs, ensuring clean FSP initialization with P2P-capable DMA configuration.

The message also demonstrates a mature engineering mindset: it documents not just the solution, but the reasoning, the failed attempts, the assumptions that were corrected, and the exact conditions required for success. It ends with a clear plan for the next reboot and a list of files that implement the fix. This is the kind of message that saves future engineers days of debugging—it captures not just what was done, but what was learned.

For anyone deploying Blackwell GPUs in environments with IOMMU, this message is a cautionary tale and a practical guide. The Blackwell FSP is not forgiving, and the only safe way to configure IOMMU identity domains is before the driver loads. The modprobe hook technique documented here is the definitive solution, and the understanding of the FSP locking behavior is essential knowledge for anyone working with NVIDIA's latest architecture.