The Moment the Blackwell FSP Refused: A Post-Mortem of IOMMU Identity Domains on NVIDIA Blackwell GPUs

In any sufficiently complex systems engineering effort, there comes a moment when a carefully constructed plan collides with an unyielding hardware reality. Message <msg id=6316> in this opencode session captures exactly such a moment — the instant when the assistant's multi-step strategy to enable peer-to-peer DMA on NVIDIA Blackwell GPUs via IOMMU identity domains fails definitively, with the stark output nvidia-smi: No devices were found despite all four GPUs having successfully bound to the nvidia driver moments earlier.

This message is the climax of a long investigation spanning dozens of prior messages, and understanding it requires reconstructing the full chain of reasoning, assumptions, and technical context that led to this point.

The Context: Why IOMMU Identity Domains Matter

The broader goal of this session was to deploy large language models (specifically Qwen3.5-122B-A10B BF16) across four NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism (TP=4). For efficient multi-GPU inference, peer-to-peer (P2P) DMA between GPUs is critical — it allows GPUs to communicate directly across the PCIe bus without staging through host memory, dramatically reducing latency for operations like all-reduce.

However, the system was running under a virtualized environment with SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) enabled, which necessitates an IOMMU (Input/Output Memory Management Unit) for DMA remapping. The IOMMU was configured in its default DMA-FQ (DMA with Fine-grained Queuing) translation mode, which provides security isolation but breaks direct GPU-to-GPU P2P DMA. The nvidia driver's DmaRemapPeerMmio=1 parameter was enabled but produced incomplete IOMMU mappings — some peer pairs worked while others faulted.

The solution that the assistant had been pursuing across the previous several messages was to switch the IOMMU groups containing the four NUMA0 GPUs from DMA-FQ mode to identity mode. In identity mode, the IOMMU performs no address translation — it effectively becomes a pass-through, allowing GPUs to see each other's physical memory addresses directly. This would restore full P2P DMA capability.

The Core Problem: Timing and the Blackwell FSP

The fundamental challenge was one of timing. The IOMMU group type can only be changed when no device in the group has a driver bound. But the nvidia driver probes and binds to GPUs automatically during PCI bus enumeration at boot time, before any userspace script can intervene. The assistant had already tried several approaches:

Udev rules that would fire when a GPU appears, but these execute too late — the driver binds before the udev RUN script runs.
Unbind/rebind cycles where the GPU is removed from the nvidia driver, the IOMMU type is changed, and then the driver is re-triggered. This failed because the Blackwell GPU's Firmware Security Processor (FSP) enters a corrupted state after unbind and cannot be reinitialized without a hardware-level power cycle.
Secondary Bus Reset (SBR) and CXL bus reset, which the assistant tested extensively in messages <msg id=6304> through <msg id=6315>. None of these software-initiated resets could clear the FSP's locked state. The one sequence that had worked earlier was: remove the GPU from the PCI bus entirely → SBR the parent PCIe bridge → rescan the PCI bus. In that sequence, the GPU would reappear as a fresh device and the nvidia driver would auto-probe it successfully. But in that successful case, the nvidia driver bound before the assistant could set the IOMMU group to identity mode — the driver auto-probed during the rescan.

The Message: A Carefully Orchestrated Attempt

Message <msg id=6316> represents the assistant's most refined attempt to solve this timing problem. The strategy was elegant: prevent nvidia from auto-probing during the PCI rescan by using a modprobe blacklist, then set the IOMMU groups to identity mode while the GPUs were driverless, and finally remove the blacklist and load nvidia manually.

The message opens with the assistant's reasoning:

All 4 GPUs back with no driver, and identity domains already set (persisted from earlier). The blacklist prevented nvidia from autoloading. Now the critical step: remove blacklist and load nvidia with clean GSP + identity domains.

This is the culmination of the previous message's work. In <msg id=6315>, the assistant had executed the full remove → SBR → rescan cycle with the nvidia blacklist in place. The result (partially visible at the end of that message) showed all four GPUs reappeared with no driver bound — the blacklist had worked. And crucially, the IOMMU groups already showed type=identity because the kernel had persisted that setting from the earlier manual configuration.

Now in <msg id=6316>, the assistant executes the payoff: a bash script that:

Confirms the identity domains — all four GPUs show type=identity in their IOMMU groups (groups 28, 61, 42, and 10 respectively).
Removes the blacklist — deletes /etc/modprobe.d/temp-nvidia-blacklist.conf.
Loads the nvidia driver — modprobe nvidia exits with code 0 (success).
Verifies driver binding — all four GPUs now show driver=nvidia.
Loads the remaining nvidia submodules — nvidia_modeset, nvidia_uvm, nvidia_drm.
Checks nvidia-smi — the critical moment: No devices were found. The output is devastating in its brevity. The driver loaded successfully. The driver bound to all four GPUs. But nvidia-smi cannot see them. The dmesg output (truncated in the message but fully visible in the subsequent <msg id=6317>) shows the familiar FSP initialization failure: RmInitAdapter failed! (0x62:0xffff:2142).

The Assumptions That Proved Wrong

This message reveals several assumptions that, while reasonable, turned out to be incorrect:

Assumption 1: The remove → SBR → rescan sequence resets the FSP. This was the critical assumption. Earlier in the session, the assistant had successfully recovered GPUs using this sequence, and the GPUs worked fine afterward. But as the assistant realizes in <msg id=6318>: "This means the earlier 'successful' resets were happening with fresh GSP state (from boot), not because SBR cleaned it." The GPUs had only worked after a full cold boot, not because the SBR actually reset the FSP. The assistant was conflating correlation with causation.

Assumption 2: The IOMMU group type persists correctly across device removal and rescan. The groups did persist with identity type, which was encouraging. But the assistant assumed this persistence was benign — that the identity mode would be compatible with the nvidia driver's initialization sequence. In reality, the Blackwell FSP apparently requires specific DMA mappings set up by the kernel's DMA API during driver initialization, and identity mode breaks this initialization path.

Assumption 3: Loading the driver after identity domains are set would work identically to loading it with DMA-FQ. The assistant assumed that the IOMMU mode was transparent to the nvidia driver — that the driver would initialize the GPU the same way regardless of whether the IOMMU was translating or pass-through. The FSP boot failure with error code 0x177 (discovered later in the segment's chunk summary) proves this assumption wrong: the FSP firmware explicitly depends on DMA translation being active during its boot sequence.

Assumption 4: The blacklist approach would produce a clean driver initialization. The assistant had carefully prevented nvidia from auto-probing, set identity domains, and then manually loaded the driver. The modprobe nvidia exit: 0 suggested success. But the driver's return code only indicates whether the module loaded, not whether it successfully initialized all devices. The actual failure happened deeper in the initialization path, invisible to modprobe.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Linux PCI subsystem architecture: How PCI devices are enumerated, how drivers bind, the role of /sys/bus/pci/devices/, and the remove/rescan interface.
IOMMU concepts: DMA remapping, translation vs. identity domains, the /sys/kernel/iommu_groups/ interface, and how IOMMU configuration affects device DMA.
NVIDIA driver architecture: The modprobe blacklist mechanism, the nvidia driver module stack (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm), and how nvidia-smi discovers devices.
Blackwell GPU architecture specifics: The Firmware Security Processor (FSP), its role in GPU initialization, and its resistance to software-initiated resets — a security feature that became a significant obstacle.
PCIe reset mechanisms: FLR (Function Level Reset), SBR (Secondary Bus Reset), and CXL bus reset, and their varying effectiveness on different hardware.
The earlier session history: The assistant's prior attempts with udev rules, unbind/rebind cycles, and the discovery that SBR only works from a clean PCI-removed state.

Output Knowledge Created

This message produced several critical pieces of knowledge:

The blacklist approach works mechanically — it is possible to prevent nvidia from auto-probing during PCI rescan, set IOMMU identity domains, and then load the driver manually. The driver binds successfully.
But identity domains break Blackwell GPU initialization — despite successful driver binding, the FSP cannot complete its boot sequence when the IOMMU is in identity mode. The error 0x62:0xffff:2142 (and the underlying 0x177 error code) is the signature of this incompatibility.
The FSP state is sticky across SBR — even with the GPU removed from the OS and the parent bridge reset, the FSP retains its corrupted state. Only a full power cycle (AC power off/on) can clear it.
The nvidia driver's modprobe exit code is not a reliable indicator of device initialization success — exit: 0 from modprobe does not mean the GPUs are usable.
A new boundary condition was discovered — the Blackwell FSP requires DMA translation mode during its boot sequence, making IOMMU identity domains fundamentally incompatible with these GPUs regardless of timing.

The Thinking Process Visible in the Message

The assistant's reasoning in this message is structured as a classic scientific experiment: state the hypothesis, describe the procedure, execute it, and report the results. The opening line — "All 4 GPUs back with no driver, and identity domains already set (persisted from earlier)" — shows the assistant verifying the preconditions before proceeding. The phrase "Now the critical step" signals awareness that this is a make-or-break moment.

The bash script itself is meticulously structured with echo statements that serve as progress markers, each one a checkpoint that the assistant would read in the output to understand where things went wrong. The sleep 8 after modprobe nvidia shows the assistant allowing ample time for driver initialization — this is not a race condition issue.

The decision to check nvidia-smi last, after confirming driver binding, shows the assistant understands the layered nature of GPU initialization: driver binding is necessary but not sufficient for device discovery. The empty output from nvidia-smi is the definitive answer the assistant was seeking.

What makes this message particularly poignant is what follows. In the very next message (<msg id=6317>), the assistant immediately checks dmesg to diagnose the failure, finding the familiar FSP error. And in <msg id=6318>, the assistant has the crucial insight: "This means the earlier 'successful' resets were happening with fresh GSP state (from boot), not because SBR cleaned it." The entire approach collapses under this realization.

The Broader Significance

This message represents a hard boundary in the systems engineering space. The assistant had explored every software-level option: udev rules, unbind/rebind, SBR, CXL bus reset, multiple SBR cycles with extended hold times, and finally the blacklist approach. All failed against the same hardware limitation: the Blackwell FSP cannot be reset by software and requires DMA translation mode.

The chunk summary for this segment confirms the final verdict: "Per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs — the approach cannot work regardless of timing." The only remaining option for P2P would be if the nvidia driver's DmaRemapPeerMmio=1 parameter could be made to work correctly (it was already enabled but produced incomplete mappings).

The session ultimately pivoted to accepting the DMA-FQ configuration and optimizing within those constraints. The MTP (Multi-Token Prediction) speculation enabled in the previous session survived the reboot and continued to provide 12-45% throughput improvement — a significant consolation prize. But the dream of full P2P DMA on Blackwell under SEV-SNP was definitively dead.

Conclusion

Message <msg id=6316> is a document of failure — but failure of the most instructive kind. It represents the moment when a carefully reasoned multi-step plan, built on the best available understanding of the system, collides with an undocumented hardware constraint. The assistant's methodology was sound: isolate variables, control timing, verify preconditions, and test incrementally. The blacklist approach was creative and technically correct. It simply couldn't overcome the Blackwell FSP's requirement for DMA translation mode during boot.

For anyone working with NVIDIA Blackwell GPUs in virtualized environments with IOMMU enabled, this message and its surrounding context provide a critical data point: identity IOMMU domains are not an option for restoring P2P DMA. The FSP's dependency on DMA translation is a hard architectural constraint that no amount of software ingenuity can circumvent. The path forward lies elsewhere — perhaps in fixing the nvidia driver's DmaRemapPeerMmio implementation, or in finding alternative communication paths like NVLink if available.

In the end, the most valuable output of this message is not the result itself but the knowledge it creates: a clear, reproducible demonstration of a fundamental incompatibility between Blackwell GPUs and IOMMU identity domains, saving future engineers from pursuing the same dead end.