The Strategic Retreat: Restoring GPUs After a Failed IOMMU Identity Experiment on Blackwell

Introduction

In the course of a deeply technical infrastructure debugging session spanning multiple days, a single message can serve as a turning point — a moment when the engineer steps back, acknowledges the limitations of the current approach, and makes a deliberate choice to regroup. Message [msg 6403] in this opencode session is exactly such a moment. After a prolonged and increasingly sophisticated attempt to restore GPU Peer-to-Peer (P2P) DMA capability under an IOMMU-enabled Linux system with NVIDIA Blackwell RTX PRO 6000 GPUs, the assistant confronts a hard reality: the GPU's Firmware Security Processor (FSP) has been corrupted by a prior nvidia driver initialization, and no amount of Secondary Bus Reset (SBR) or software manipulation can clear it. The only path forward is to restore the system to a working state, defer the experiment to the next boot cycle, and keep the production inference service running.

This article examines that message in detail: what it contains, why it was written, the reasoning that led to it, the decisions embedded within it, and the knowledge it both consumes and produces.

The Message: A Controlled Restoration

The subject message reads in full:

[assistant] Now, for the current session, the GPUs have corrupted GSP and we need a reboot to test the modprobe hook properly. But let me first restore the GPUs to working state (without identity domains) so SGLang can run: [bash] ssh root@10.1.2.6 'set -e # Temporarily disable the modprobe hook so nvidia loads normally mv /etc/modprobe.d/nvidia-iommu-identity.conf /etc/modprobe.d/nvidia-iommu-identity.conf.bak

GPUS="0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0" BRIDGES="0000:00:01.1 0000:10:01.1 0000:60:01.1 0000:70:01.1"

# Unbind + unload for gpu in $GPUS; do echo "$gpu" > /sys/bus/pci/drivers/nvidia/unbind 2>/dev/null || true; done rmmod nvidia_drm 2>/dev/null; rmmod nvidia_uvm 2>/dev/null; rmmod nvidia_modeset 2>/dev/null; rmmod nvidia 2>/dev/null

# Remove + SBR for gpu in $GPUS; do echo 1 > /sys/bus/pci/devices/$gpu/remove 2>/dev/null || true; done sleep 2 for bridge in $BRIDGES; do bc=$(setpci -s $bridge BRIDGE_CONTROL); nv=$(printf "%04x" $((0x$bc | 0x40))) setpci -s $bridge BRIDGE_CONTROL=$nv; sleep 0.5; setpci -s $bridge BRIDGE_CONTROL=$bc done sleep 15

# Block vfio-pci echo "10de 2bb5" > /sys/bus/pci/drivers/vfio-pci/remove_id 2>/dev/null || true

# Rescan (nvidia loads normally without hook) echo 1 > /sys/bus/pci/rescan sleep 15

# Restore the hook for next boot mv /etc/modprobe.d/nvidia-iommu-identity.conf.bak /etc/modprobe.d/nvidia-iommu-identity.conf

# Load supplementary modules modprobe nvidia_modeset 2>/dev/null; modprobe nvidia_uvm 2>/dev/null; modprobe nvidia_drm 2>/dev/null sleep 3

# Re-register vfio-pci echo "10de 2bb5" > /sys/bus/pci/drivers/vfio-pci/new_id 2>/dev/null || true

nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader 2>&1' 0, 00000000:01:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB 1, 00000000:11:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB 2, 00000000:61:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB 3, 00000000:71:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB

The message is deceptively simple on its surface: a bash script that restores four NVIDIA Blackwell GPUs to working order after a failed experiment. But beneath that surface lies a wealth of reasoning, decision-making, and accumulated domain knowledge.

Context: The IOMMU Identity Domain Quest

To understand why this message was written, one must understand the journey that preceded it. The system in question is a Proxmox host with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs on NUMA node 0) and a SEV-SNP VM (4 GPUs on vfio-pci). The LXC container runs SGLang serving a Qwen3.5-122B-A10B BF16 model with tensor parallelism across 4 GPUs.

The core problem is that IOMMU translation mode, while necessary for SEV-SNP security isolation, breaks GPU P2P DMA — the direct GPU-to-GPU data transfers that are critical for tensor-parallel inference performance. The nvidia driver's DmaRemapPeerMmio=1 parameter attempts to work around this but produces incomplete IOMMU mappings, with some peer pairs working and others faulting.

The assistant had been pursuing a creative solution: setting individual IOMMU groups to "identity" mode (which bypasses translation) for just the NUMA0 GPUs, while leaving the VFIO GPUs in translation mode. This would theoretically restore P2P DMA for the inference GPUs while maintaining security isolation for the VM.

The approach evolved through several iterations. Initially, a systemd service attempted to set identity domains after boot, but it ran too late — the nvidia driver had already initialized the GPUs and locked their FSP state. The assistant then devised a more elegant solution: a modprobe install hook that would execute a script before the nvidia module loaded, setting identity domains while the GPUs were still unclaimed. This hook was created in messages [msg 6393] through [msg 6398], tested in [msg 6399], and found to work — the identity domains were set correctly — but the nvidia driver still failed with GSP initialization errors (error code 0x177 and "RmInitAdapter failed" messages).

The Critical Realization

Message [msg 6401] contains the key insight: the SBR (Secondary Bus Reset) does not clear the Blackwell GSP firmware state once the nvidia driver has initialized it. The assistant writes:

"Wait... I've been assuming the SBR clears the GSP because it worked once before. But looking more carefully, the successful recovery earlier in the previous session happened when the GPUs had NEVER been bound to nvidia before the SBR — they were from a completely fresh boot. In THIS boot, the nvidia driver already loaded and initialized the GPUs (the identity service ran AFTER nvidia at boot due to ordering), which locked the GSP. Now no amount of SBR can clear it."

This is a profound realization. The Blackwell architecture introduces a Firmware Security Processor (FSP) that manages GPU initialization and security-critical operations. Once the nvidia driver initializes the FSP during its first probe of the GPU, the FSP enters a state that cannot be reset through software mechanisms like SBR, Function Level Reset (FLR), or even CXL bus resets. The only way to get a clean FSP is a full power-on reset — i.e., a system reboot.

This means the modprobe hook approach is correct in theory but cannot be tested in the current boot session. The GPUs are "poisoned" by the earlier nvidia initialization that happened before the identity domain service could run. The modprobe hook, which runs at the very next modprobe invocation, will only take effect after a reboot when the GPUs have clean FSP state from cold power-on.

Decisions Made in This Message

Message [msg 6403] embodies several deliberate decisions:

Decision 1: Defer rather than force. The assistant could have chosen to reboot immediately to test the modprobe hook. Instead, it decides to restore the GPUs to working order first, keeping the production SGLang service running. This prioritizes operational stability over experimental progress.

Decision 2: Preserve the modprobe hook for the next boot. The script temporarily moves the hook configuration aside (nvidia-iommu-identity.conf.bak), performs the restoration without it, then moves it back. This ensures the hook will be active on the next reboot without affecting the current recovery.

Decision 3: Use the proven SBR recovery sequence. The restoration script follows the same PCI device removal, SBR, and rescan pattern that has been refined over many iterations in this session. It's a well-understood, reliable procedure at this point.

Decision 4: Accept the current limitations. The assistant implicitly accepts that P2P DMA will remain disabled (NCCL_P2P_DISABLE=1) for the current session, and that MTP (Multi-Token Prediction) speculation — which provides 12-45% throughput improvement — will serve as the primary optimization until the next boot test.

Assumptions and Their Validity

The message operates under several assumptions:

Assumption 1: The modprobe hook will work after reboot. This is the central hypothesis being preserved. The assistant assumes that with a clean FSP from power-on, setting identity domains before nvidia loads will allow the driver to initialize successfully. This assumption was later tested (in the subsequent chunk) and found to be incorrect — Blackwell's FSP fundamentally requires DMA translation mode during its boot sequence, and identity mode causes a fatal error regardless of timing. The approach was ultimately abandoned.

Assumption 2: The SBR recovery sequence is reliable. This assumption is well-validated by this point in the session — the assistant has executed this sequence multiple times with consistent success.

Assumption 3: Moving the modprobe config file is safe. The script temporarily renames the configuration file, relying on the fact that the modprobe hook won't be triggered during the recovery (since nvidia is loaded via PCI rescan modalias, not explicit modprobe). This is a reasonable assumption given the assistant's understanding of the Linux PCI driver model.

Assumption 4: The VFIO GPUs are unaffected. The script only manipulates the NUMA0 GPUs (01:00.0, 11:00.0, 61:00.0, 71:00.0). The 4 VFIO GPUs on NUMA1 (81:00.0, 91:00.0, e1:00.0, f1:00.0) are left untouched, and the assistant verifies them in the subsequent message [msg 6404].

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

Linux PCI subsystem: Understanding of PCI device removal (/sys/bus/pci/devices/*/remove), driver unbinding, bus rescan, and how the kernel's driver model works with modalias-based module loading.

IOMMU concepts: Understanding of DMA translation modes (DMA-FQ vs identity), IOMMU groups, and how they affect PCI device DMA operations. The DMA-FQ (DMA with Fine-grained Translation) mode provides full IOMMU translation with device isolation, while identity mode bypasses translation entirely.

NVIDIA GPU architecture (Blackwell-specific): Knowledge of the Firmware Security Processor (FSP), its role in GPU initialization, and the implications of error codes like 0x177 and "RmInitAdapter failed." The Blackwell architecture's FSP behavior is a critical piece of knowledge that the assistant acquired through experimentation during this session.

PCIe bridge operations: Understanding of Secondary Bus Reset (SBR) via the Bridge Control register, including the bit manipulation (0x40 = SBR bit) and the timing requirements for the reset to complete.

The modprobe install hook mechanism: The install directive in modprobe configuration files allows replacing the default module loading behavior with a custom command. This is an advanced Linux module loading technique.

SGLang and inference serving: The context of why these GPUs need to be operational — serving a Qwen3.5-122B-A10B BF16 model with tensor parallelism — and why P2P DMA matters for performance.

Output Knowledge Created

This message produces several valuable outputs:

A validated recovery procedure. The SBR-based GPU recovery sequence is demonstrated to work reliably, even after the GPUs have been through multiple driver initialization cycles. This is a reusable procedure for any Blackwell GPU system that needs to recover from a failed driver state.

A preserved experimental configuration. The modprobe hook is saved and will be active on the next boot, enabling the identity domain test under the correct conditions (clean FSP).

Confirmation of Blackwell FSP persistence. The message implicitly confirms that the Blackwell FSP state persists across SBR, which is a significant architectural finding. This knowledge informs future troubleshooting: if the nvidia driver has ever initialized a Blackwell GPU in a given boot session, software resets cannot fully recover the GPU — only a power cycle will suffice.

A stable production state. The final nvidia-smi output confirms all 4 NUMA0 GPUs are online with full memory (97887 MiB each), ready for SGLang to resume serving.

The Thinking Process Visible in the Reasoning

The message's opening line reveals the assistant's thought process: "Now, for the current session, the GPUs have corrupted GSP and we need a reboot to test the modprobe hook properly." This is a statement of acceptance — the assistant has concluded that the current boot session cannot yield the desired result and is pivoting to a strategy of preservation and deferral.

The structure of the bash script itself reveals careful thinking about state management. The script:

Backs up the modprobe config before making any changes, ensuring the experimental configuration is preserved.
Unbinds and unloads in the correct order — first unbinding individual GPUs from the nvidia driver, then removing the nvidia module stack in dependency order (drm → uvm → modeset → nvidia).
Removes PCI devices from the bus, making them invisible to the kernel.
Performs SBR on the upstream bridges, which should reset the PCIe link and GPU internal state.
Blocks vfio-pci from claiming the GPUs during rescan by removing the device ID from vfio-pci's new_id table.
Rescans the PCI bus with the modprobe hook temporarily disabled, allowing nvidia to load normally.
Restores the modprobe hook after the GPUs are successfully bound.
Loads supplementary modules in the correct order.
Re-registers vfio-pci for the VFIO GPUs. This sequence demonstrates a deep understanding of the Linux PCI driver lifecycle and the dependencies between the various nvidia kernel modules. The careful ordering, the error suppression (2>/dev/null || true), and the timing delays (sleep 15 for SBR and rescan) all reflect accumulated experience from previous failed attempts.

Conclusion

Message [msg 6403] is a masterclass in controlled retreat. It acknowledges failure without abandoning the experimental thread, preserves the infrastructure for future testing, and restores the system to a known good state. The assistant's reasoning — that the Blackwell FSP cannot be cleared by software reset once initialized — represents a significant piece of architectural knowledge gained through painful experimentation.

The message also demonstrates a crucial engineering virtue: knowing when to stop pushing against a wall and instead set up the conditions for a different approach. By deferring the modprobe hook test to the next boot cycle, the assistant avoids wasting time on a fundamentally unrecoverable state and instead focuses on what can be achieved: a stable, running inference service with MTP speculation providing meaningful throughput gains.

In the larger narrative of this session, this message marks the end of the IOMMU identity domain experiment. The subsequent chunk reveals that even after reboot, the approach failed — Blackwell's FSP fundamentally requires DMA translation mode during boot. But that discovery could only be made because this message first restored the system to a working state, proving that sometimes the most productive thing to do is to step back, clean up, and try again from a clean slate.