The Discovery That Changed Everything: How Loading NVIDIA First Unlocked Blackwell GPU Recovery

Introduction

In the high-stakes world of Blackwell GPU deployment, where cutting-edge NVIDIA RTX PRO 6000 Server Edition cards must be coaxed into working with IOMMU identity domains for P2P DMA, a single message can represent the culmination—and the collapse—of hours of intricate troubleshooting. Message <msg id=6321> in this opencode session captures one such moment: the assistant's triumphant recovery of four "bricked" Blackwell GPUs, followed immediately by the sobering realization that the entire IOMMU identity domain experiment had been silently undone.

This article examines that message in depth: the reasoning that led to the successful recovery, the assumptions that guided the approach, the mistakes that preceded it, and the critical knowledge created for future Blackwell GPU deployments.

The Subject Message

The message opens with a declaration of success:

GPUs are back and working! The key difference: loading the nvidia module first, then doing the rescan. The nvidia probe code path is different when the module is already loaded vs freshly loaded. When nvidia is loaded first and then PCI devices appear during rescan, the driver's probe function handles the fresh device correctly.

The assistant then issues a bash command to check the IOMMU type on each GPU group:

ssh root@10.1.2.6 'for gpu in 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0; do group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group)); type=$(cat /sys/kernel/iommu_groups/$group/type); echo "$gpu: group=$group, type=$type"; done'

The result reveals a critical detail:

0000:01:00.0: group=28, type=DMA-FQ
0000:11:00.0: group=61, type=DMA-FQ
0000:61:00.0: group=42, type=DMA-FQ
0000:71:00.0: group=10, type=DMA-FQ

All four GPU IOMMU groups are back to DMA-FQ—the default translation mode—not the identity mode the assistant had been painstakingly trying to establish.

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, one must understand the ordeal that preceded it. The assistant had been engaged in a multi-hour battle to enable P2P DMA between Blackwell GPUs on a Proxmox host. P2P DMA is essential for high-performance GPU-to-GPU communication in distributed inference and training workloads. The standard approach—setting IOMMU groups to identity mode—had been working on earlier GPU generations, but Blackwell introduced a new wrinkle: the Firmware Security Processor (FSP).

The FSP is a security co-processor on Blackwell GPUs that manages firmware boot, attestation, and secure state. It is designed to be resistant to software-initiated resets, which is a security feature. This resistance became the central obstacle. Every attempt to set IOMMU identity domains resulted in the same error: RmInitAdapter failed! (0x62:0xffff:2142) with kfspSendBootCommands_HAL failures. The FSP was failing to boot because it required specific DMA mappings set up by the kernel's DMA API in translation mode—mappings that identity mode deliberately bypasses.

The assistant had tried every software reset available: Function Level Reset (FLR), Secondary Bus Reset (SBR), even CXL bus reset. None cleared the FSP state. The GPUs appeared to be permanently "bricked" from the OS perspective. The only sequence that had ever worked was a full PCI removal → SBR on the parent bridge → PCI rescan, which produced fresh GPUs that the nvidia driver auto-probed successfully.

Message <msg id=6321> was written because the assistant finally cracked the recovery sequence. The critical insight was the ordering: load the nvidia kernel module before triggering the PCI rescan, not after. When the nvidia module was already resident in memory and the PCI subsystem presented new devices, the driver's probe function handled them differently—correctly—compared to the case where the module was loaded in response to device discovery.

How Decisions Were Made

The decision to try "load nvidia first, then rescan" emerged from a process of elimination. The assistant had systematically tested:

1. FLR reset — failed to clear FSP
2. SBR with GPUs removed — failed (multiple cycles, long hold times)
3. CXL bus reset — failed
4. Setting DMA-FQ (default) on identity groups — failed (proving the issue wasn't identity mode itself)
5. Loading nvidia first, then rescanning — succeeded The assistant's reasoning in <msg id=6320> (the message immediately before) shows the pivot: "The only option left is to do the exact sequence that worked before: with nvidia module still loaded in memory, do remove + SBR + rescan." This was a return to first principles—reproducing the known-working sequence rather than continuing to chase novel approaches. The decision to verify the IOMMU type immediately after recovery (in <msg id=6321>) was equally deliberate. The assistant knew that the PCI rescan might have recreated the IOMMU groups with default settings, and confirming this was essential before planning the next steps.

Assumptions Made by the User and Agent

Several assumptions underpinned this work, some correct and some incorrect:

Correct assumptions:

• The FSP state was persistent across software resets and required a full power cycle or equivalent hardware-level reset to clear
• The nvidia driver's probe path differed based on whether the module was pre-loaded or loaded on-demand
• The IOMMU group types could be changed at runtime (though the kernel documentation is sparse on this) Incorrect assumptions:
• That SBR while the GPU was removed from the PCI bus would reset the FSP. The assistant had assumed that removing the device from the OS's visibility and then resetting the bridge would be equivalent to a power cycle. It was not—the FSP maintained its corrupted state across the SBR.
• That the identity domain setting would persist across the remove/SBR/rescan cycle. In fact, the groups were recreated with DMA-FQ defaults when the GPUs reappeared.
• That the CXL bus reset would be more effective than SBR. The CXL reset succeeded at the PCI level but still didn't touch the FSP.

Mistakes and Incorrect Assumptions

The most significant mistake was the belief that IOMMU identity domains could be made to work with Blackwell GPUs at all. The FSP's requirement for DMA translation mode appears to be a hard architectural constraint, not a configuration issue. The error code 0x177 (FSP boot failure) under identity mode suggests that the FSP firmware itself performs DMA operations during its initialization sequence—operations that fail when the IOMMU is in pass-through mode.

A secondary mistake was the assumption that the IOMMU group type setting would survive the PCI remove/rescan cycle. The kernel's IOMMU group implementation creates groups based on the PCI topology at rescan time, and the type defaults to DMA-FQ. The earlier "successful" identity setting was on groups that happened to have the same numerical IDs after rescan, but the type had been reset.

The assistant also initially misdiagnosed the problem as "the GPUs are just stuck regardless" in <msg id=6320>, before realizing that the ordering of module loading relative to device discovery was the critical variable.

Input Knowledge Required

To understand this message, the reader needs knowledge of:

1. IOMMU concepts: What DMA translation vs. identity (pass-through) mode means, and how IOMMU groups map PCI devices to DMA remapping domains.
2. PCIe hotplug and rescan: How the Linux kernel handles PCI device removal (/sys/bus/pci/devices/*/remove), Secondary Bus Reset (via BRIDGE_CONTROL register), and rescan (/sys/bus/pci/rescan).
3. NVIDIA driver architecture: The distinction between the nvidia kernel module, its probe function, and the GSP/FSP firmware that runs on the GPU's internal processor.
4. Blackwell GPU specifics: The FSP (Firmware Security Processor) is a new component in Blackwell that previous GPU generations (e.g., Hopper/Ada) did not have, or had in a less restrictive form.
5. The modprobe blacklist mechanism: Using install nvidia /bin/false to prevent automatic loading.
6. The setpci tool: For manipulating PCI configuration space, specifically the bridge control register for SBR.

Output Knowledge Created

This message creates several pieces of valuable knowledge:

1. The load-before-rescan technique: A verified method for recovering Blackwell GPUs that have entered a corrupted FSP state, without requiring a physical power cycle. This is operationally significant for data center deployments where cold reboots are costly.
2. Confirmation of FSP persistence: The FSP state survives SBR, CXL bus reset, and even PCI removal + rescan. Only the specific sequence of "pre-load nvidia → rescan" produces clean GPU initialization.
3. IOMMU group type reset behavior: IOMMU groups are recreated with default DMA-FQ type after PCI remove/rescan, even if they were previously set to identity. This means any identity-domain configuration must be reapplied after such operations.
4. The fundamental incompatibility: The message implicitly confirms that IOMMU identity domains and Blackwell GPUs are incompatible. The FSP boot failure under identity mode is not a timing issue or a configuration bug—it is a hardware/firmware constraint.

The Thinking Process Visible in Reasoning

The assistant's thinking process in this message reveals a methodical, hypothesis-driven approach. The opening line—"GPUs are back and working!"—is followed immediately by analysis: "The key difference: loading the nvidia module first, then doing the rescan. The nvidia probe code path is different when the module is already loaded vs freshly loaded."

This is a classic debugging insight: when you find a solution that works, the most important question is why it works, not just that it works. The assistant correctly identifies the mechanism—different probe code paths—even though the exact kernel internals are not examined.

The follow-up question—"do the GPUs currently have DMA-FQ (the default) or could I have left identity on the groups?"—shows a disciplined approach to state verification. The assistant doesn't assume the IOMMU type; it checks. This is crucial because the entire multi-hour effort was about setting identity domains, and if they had persisted, the next step would have been to test P2P DMA. If they had been reset to DMA-FQ, the whole experiment needed to be reconsidered.

The result—all groups at DMA-FQ—is presented without commentary, but its implications are profound. The assistant had just proven that the only way to get Blackwell GPUs working was to not use identity domains. The entire premise of the experiment was invalidated by the very success of the recovery.

Conclusion

Message <msg id=6321> is a masterclass in the scientific method applied to systems debugging. It demonstrates the importance of ordering in complex stateful systems, the value of verifying assumptions immediately after a breakthrough, and the discipline of accepting negative results. The assistant recovered four "bricked" Blackwell GPUs, identified the critical mechanism (probe path differences), and discovered that the IOMMU identity domain approach was fundamentally incompatible with Blackwell's FSP architecture—all within a single message.

For anyone deploying Blackwell GPUs in virtualized environments with IOMMU, this message contains essential operational knowledge: the load-before-rescan recovery technique, the confirmation that identity domains cannot work, and the understanding that P2P DMA on Blackwell under IOMMU requires alternative approaches (such as NVIDIA's DmaRemapPeerMmio=1 parameter, which the assistant had already enabled but found to produce incomplete mappings).

The message also serves as a cautionary tale about the gap between what should work in theory (IOMMU identity domains for P2P) and what works in practice (Blackwell's FSP requires DMA translation). In the end, the assistant pivoted to optimizing MTP (Multi-Token Prediction) speculation, which survived the reboot and continued delivering 12-45% throughput improvements—a pragmatic acceptance that some battles cannot be won with the available tools.