The Moment of Clarity: Recovering Stuck Blackwell GPUs After IOMMU Experiments

In the high-stakes world of GPU infrastructure debugging, few moments are as dramatic as the one captured in message 6320 of this opencode session. After hours of painstaking experimentation with IOMMU identity domains, PCI resets, and Blackwell GPU firmware, the assistant arrives at a critical realization: the GPUs are not broken because of the IOMMU configuration — they are broken because of the very act of unbinding and resetting them. This message represents the turning point where systematic debugging finally yields a working recovery sequence, revealing deep truths about the relationship between the nvidia driver, the Blackwell Firmware Security Processor (FSP), and the PCI subsystem.

The Context: A Quest for P2P DMA

To understand why this message matters, we must first understand what led to it. The assistant had been working on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between a LXC container and a VM. Four GPUs (on NUMA0) were bound to the nvidia driver for serving the Qwen3.5-122B-A10B model with SGLang, while the other four were on vfio-pci for a SEV-SNP VM. The problem was that P2P (peer-to-peer) DMA between GPUs was broken under the SEV-SNP IOMMU configuration, causing NCCL to hang.

The assistant's strategy was to set the IOMMU groups for the NUMA0 GPUs to "identity" mode — a configuration where DMA translations pass through without remapping, which should allow P2P transfers to work correctly. The challenge was timing: the identity mode had to be set before the nvidia driver loaded, because the driver's initialization of the Blackwell FSP (a security coprocessor embedded in the GPU) requires specific DMA mappings that identity mode might disrupt.

The Descent Into Failure

The preceding messages ([msg 6308] through [msg 6319]) document a painful debugging spiral. The assistant tried:

Unbinding GPUs from nvidia, removing them from PCI, issuing Secondary Bus Reset (SBR) on parent bridges, then rescanning — this worked initially but left the GPUs in a state where the nvidia driver could no longer initialize them.
CXL bus reset — a more aggressive reset mechanism that also failed to clear the FSP state.
Multiple SBR cycles with extended hold times — holding the reset signal for 2 seconds per cycle, repeating three times, with a 15-second final wait. Still no success.
Reverting IOMMU groups from identity back to DMA-FQ — to test whether identity mode itself was causing the FSP boot failure. Every attempt returned the same result: nvidia-smi reported "No devices were found," and the kernel log showed the dreaded RmInitAdapter failed! (0x62:0xffff:2142) error — the FSP was refusing to boot.

The Critical Realization

Message 6320 opens with a stark admission: "Also fails! So the issue is NOT the identity domain — the GPUs are just stuck regardless. The SBR while GPUs were removed doesn't clear the FSP state either. The GPUs have been fundamentally corrupted from the very first unbind."

This is the breakthrough moment. The assistant had been operating under the assumption that the identity IOMMU mode was the root cause of the FSP boot failure. But the test just performed — reverting to DMA-FQ and loading nvidia — proved otherwise. The GPUs failed to initialize even with the default IOMMU configuration. Something more fundamental was wrong.

The reasoning here is exemplary of good debugging: the assistant had formulated a hypothesis (identity mode breaks FSP boot), tested it (revert to DMA-FQ), and the hypothesis was falsified. This forced a re-examination of the entire sequence of events.

The assistant then reconstructs the timeline: "The only option left is to do the exact sequence that worked before: with nvidia module still loaded in memory, do remove + SBR + rescan."

The Recovery Sequence

The bash script that follows is the culmination of hours of trial and error. Let's examine its logic step by step:

Remove GPUs from PCI: The four NUMA0 GPUs (at 0000:01:00.0, 0000:11:00.0, 0000:61:00.0, and 0000:71:00.0) are removed from the PCI bus by writing 1 to their remove sysfs files. At this point, the nvidia module is not loaded — the assistant had unloaded it in previous steps.
Secondary Bus Reset: The parent PCI bridges (at 0000:00:01.1, 0000:10:01.1, 0000:60:01.1, and 0000:70:01.1) are each issued an SBR by setting the BRIDGE_CONTROL register's secondary bus reset bit (0x40), holding it for 2 seconds, then clearing it. This is a more aggressive reset than FLR (Function Level Reset) because it resets the entire bus segment below the bridge.
Load nvidia module BEFORE rescan: This is the critical innovation. In all previous attempts, the assistant either loaded nvidia after rescan (allowing the driver to auto-probe the fresh GPUs) or tried to set identity domains before loading. Here, the nvidia module is loaded while the GPUs are still absent from the PCI bus. This means the driver initializes its internal state without any devices attached.
Rescan PCI bus: Writing 1 to /sys/bus/pci/rescan triggers a full PCI bus enumeration. The freshly discovered GPUs are now probed by the already-loaded nvidia driver. The result: nvidia-smi returns successfully, showing all four GPUs with their full 97,887 MiB of memory.

Why This Works: The FSP Initialization Mystery

The success of this sequence reveals something profound about the Blackwell GPU architecture. The FSP (Firmware Security Processor) is a dedicated security coprocessor embedded in the GPU that handles firmware boot, attestation, and secure state management. Unlike previous GPU generations, Blackwell's FSP is designed to be resistant to software-initiated resets — a security feature that prevents attackers from resetting the GPU to bypass firmware security checks.

The key insight is that the FSP state is not fully reset by PCI-level operations like FLR, SBR, or even CXL bus reset. When the nvidia driver binds to a GPU, it initiates a boot sequence with the FSP. If that sequence fails — for example, because the driver was unloaded mid-operation, or because IOMMU mappings changed unexpectedly — the FSP enters an error state that persists across PCI resets.

The recovery sequence works because:

Removing the GPUs from PCI disconnects the devices from the bus entirely, allowing the bridge reset to reach deeper into the hardware state.
Loading the nvidia driver first ensures that when the GPUs appear on the bus, the driver is ready to perform a clean initialization sequence from scratch.
The rescan triggers fresh PCI enumeration, which causes the driver to discover the GPUs as new devices and initialize them with a clean FSP boot sequence. The critical detail is that the nvidia driver must be loaded before the GPUs appear on the bus. If the driver loads after the GPUs are discovered, it may inherit stale state from the PCI configuration space. By loading the driver first, the assistant ensures that the driver's device probe function runs on what is effectively a "cold" device discovery.

Assumptions and Their Consequences

This message reveals several assumptions the assistant was operating under, some of which turned out to be incorrect:

Assumption 1: Identity IOMMU mode was the root cause of FSP boot failure. This was the working hypothesis for several messages, and it drove the assistant to try complex timing schemes to set identity domains before nvidia loaded. The test in message 6319 (reverting to DMA-FQ) disproved this — the GPUs still failed to initialize.

Assumption 2: SBR while GPUs are removed from the OS would fully reset the FSP. The assistant had observed earlier that "remove → SBR → rescan" produced working GPUs, but those were fresh from a cold boot. The accumulated state from multiple unbind/reset cycles had corrupted the FSP in a way that SBR alone couldn't clear.

Assumption 3: The nvidia driver could be unloaded and reloaded without side effects. This is generally true for most PCI devices, but Blackwell's FSP adds a layer of persistent state that survives driver unload. The FSP's secure boot sequence is designed to be a one-way process — once started, interruptions can leave it in an unrecoverable state until a full power cycle.

Input Knowledge Required

To fully appreciate this message, the reader needs to understand several technical domains:

PCIe topology and sysfs interface: The GPUs are behind PCI bridges, and manipulating devices via /sys/bus/pci/devices/.../remove, /sys/bus/pci/rescan, and bridge control registers requires knowledge of the PCI hierarchy.
IOMMU groups and DMA remapping: The kernel's IOMMU subsystem groups devices into DMA isolation domains, and the type sysfs file controls whether DMA translations are remapped (DMA-FQ) or pass-through (identity).
NVIDIA driver architecture: The nvidia driver consists of multiple kernel modules (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm) that must be loaded in a specific order.
Blackwell FSP/GSP firmware: The Firmware Security Processor is a new feature in Blackwell GPUs that handles secure boot and firmware management. It is separate from the GPU's main processor and has its own firmware that must be initialized by the driver.
Secondary Bus Reset mechanics: SBR is a PCIe feature that resets all devices downstream of a bridge. It is more powerful than FLR but still does not affect the GPU's internal firmware state in the way a full power cycle would.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

The recovery sequence for stuck Blackwell GPUs: Load the nvidia driver first, then rescan the PCI bus. This is now a documented procedure for recovering GPUs that fail to initialize after driver unbind/reset cycles.
The FSP state persistence problem: Blackwell's FSP state is not cleared by any software-initiated PCI reset (FLR, SBR, CXL bus). This is a significant constraint for any system management software that needs to reset GPUs without a full power cycle.
The irrelevance of IOMMU mode to the immediate failure: While identity IOMMU mode may cause FSP boot failures under certain conditions, the immediate problem was FSP corruption from prior unbind operations, not the IOMMU configuration itself.
The importance of driver-first initialization: For Blackwell GPUs, the nvidia driver must be loaded before the devices appear on the PCI bus for a clean initialization. This has implications for boot-time configuration and hot-plug scenarios.

The Thinking Process

The assistant's reasoning in this message is a textbook example of diagnostic backtracking. The sequence goes:

Observe failure: GPUs don't appear in nvidia-smi after multiple reset attempts.
Test hypothesis: Revert IOMMU to DMA-FQ to test if identity mode is the cause.
Falsify hypothesis: GPUs still fail with DMA-FQ, so identity mode is not the root cause.
Reconstruct working state: Recall that the "remove → SBR → rescan" sequence worked earlier when nvidia was still loaded.
Identify the critical variable: The difference between working and non-working states is whether nvidia was loaded before the rescan.
Execute the recovery: Load nvidia first, then rescan.
Verify success: nvidia-smi shows all four GPUs. The key insight is the realization that "the GPUs have been fundamentally corrupted from the very first unbind." This is a crucial moment of reframing — the assistant stops looking for the right IOMMU configuration and instead focuses on recovering from accumulated state corruption.

Broader Implications

This message has implications beyond this specific debugging session. It reveals that Blackwell GPUs have a fundamentally different reset behavior than previous NVIDIA generations. On Ampere or Hopper GPUs, a driver unload followed by FLR or SBR would typically be sufficient to reset the device. Blackwell's FSP adds a new layer of persistent state that changes the calculus for GPU management.

For system administrators and infrastructure engineers, this means:

Hot-unplug and replug of Blackwell GPUs requires careful sequencing: The driver must be loaded before the devices are enumerated.
FSP state corruption can persist across PCI resets: A full power cycle (AC power off/on) may be the only way to clear certain error states.
IOMMU configuration changes should be made at boot time, not dynamically: The complexity of setting identity domains after boot, combined with the FSP state persistence issue, makes runtime IOMMU reconfiguration extremely risky.

Conclusion

Message 6320 is the turning point in a long debugging saga. It represents the moment when the assistant stops chasing false hypotheses and reconstructs the one sequence that actually works. The recovery — loading the nvidia driver before rescanning the PCI bus — is elegant in its simplicity, but it was only discovered after exhausting more complex approaches.

The deeper lesson is about the Blackwell GPU architecture itself: the FSP introduces a new class of persistent state that changes how GPUs must be managed. This is not a bug but a design choice — the FSP's resistance to software reset is a security feature that prevents firmware-level attacks. But it also means that system management software must be redesigned to account for this new constraint.

For the assistant, this message represents a hard-won victory. The GPUs are back online, the SGLang service can resume, and the team has learned a critical lesson about Blackwell GPU management that will inform every future deployment. The identity IOMMU experiment may have failed to enable P2P DMA, but the knowledge gained about FSP state persistence and recovery sequences is arguably more valuable for the long-term operation of this infrastructure.