The Blackwell FSP Barrier: When IOMMU Identity Domains Meet NVIDIA's Firmware Security Processor

Introduction

In the high-stakes world of multi-GPU machine learning inference, every microsecond counts. When serving large language models across multiple GPUs, the ability for GPUs to communicate directly via Peer-to-Peer (P2P) DMA — bypassing host memory entirely — can mean the difference between 100 tok/s and 300 tok/s. This is the story of a single message in a coding session that represents the final, definitive collision between a well-intentioned system optimization and a hard hardware limitation in NVIDIA's Blackwell architecture.

Message <msg id=6300> is deceptively simple. It contains a bash script that sets four IOMMU groups to "identity" mode and loads the NVIDIA driver. Its output is devastatingly brief: "No devices were found." This message is the turning point where an entire line of investigation — spanning dozens of messages, multiple reboots, Secondary Bus Resets (SBR), GPU removal/rescan cycles, and creative udev scripting — collapses against an immovable wall: the Blackwell Firmware Security Processor (FSP) requires DMA translation mode to boot, and no software trick can change that.

The Message

The assistant writes:

Now let me set identity on the groups (GPUs should be unbound since nvidia is unloaded), then load nvidia:

```bash ssh root@10.1.2.6 'set -e GPUS="0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0"

echo "=== Check current state ===" for gpu in $GPUS; do driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none") group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group)) type=$(cat /sys/kernel/iommu_groups/$group/type) echo " $gpu: driver=$driver, group=$group, type=$type" done

echo "" echo "=== Set identity domains ===" for gpu in $GPUS; do group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group)) echo "identity" > /sys/kernel/iommu_groups/$group/type 2>&1 && echo " $gpu group=$group -> identity" || echo " $gpu group=$group -> FAILED (need unbind?)" done

echo "" echo "=== Load nvidia ===" modprobe nvidia sleep 5 modprobe nvidia_modeset; modprobe nvidia_uvm; modprobe nvidia_drm sleep 5

echo "" echo "=== nvidia-smi ===" nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader 2>&1

echo "" echo "=== dmesg nvidia ===" dmesg | grep -E "NVRM.*(failed|error|Error|Cannot)" | tail -5' ```

Output: ``` === Check current state === 0000:01:00.0: driver=none, group=28, type=identity 0000:11:00.0: driver=none, group=61, type=identity 0000:61:00.0: driver=none, group=42, type=identity 0000:71:00.0: driver=none, group=10, type=identity

=== Set identity domains === 0000:01:00.0 group=28 -> identity 0000:11:00.0 group=61 -> identity 0000:61:00.0 group=42 -> identity 0000:71:00.0 group=10 -> identity

=== Load nvidia ===

=== nvidia-smi === No devices were found ```

The output tells a story of apparent success followed by utter failure. All four GPUs are detected by the kernel, all four IOMMU groups accept the identity domain assignment, the NVIDIA modules load without error — and yet nvidia-smi reports nothing. The GPUs are invisible to the driver.

The Context: A Long Road to This Moment

To understand why this message matters, we must understand the journey that led here. The system in question is a Proxmox host with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs for inference serving) and a SEV-SNP VM (4 GPUs for confidential computing). The inference container runs SGLang serving a Qwen3.5-122B-A10B BF16 model with Tensor Parallelism across 4 GPUs.

The central problem has been P2P DMA. When GPUs communicate directly over the PCIe bus, they avoid the overhead of copying through host memory, dramatically improving multi-GPU inference throughput. But on this system, P2P DMA was broken. The root cause: an IOMMU (Input-Output Memory Management Unit) configured in DMA translation mode, which intercepts and translates all DMA transactions. When GPUs attempt P2P access, the IOMMU needs to map the remote GPU's memory into the local GPU's address space — and those mappings were incomplete or faulting.

The assistant's strategy was elegant: change the IOMMU group type from "DMA-FQ" (DMA with Fine-Grained translation) to "identity" (passthrough mode, where the IOMMU translates addresses 1:1, effectively bypassing translation). In identity mode, P2P DMA should work natively because the IOMMU isn't rewriting addresses — GPUs see each other's physical memory directly.

But changing the IOMMU group type is not a simple operation. The group's type can only be changed when no device in the group is bound to a driver. This constraint forced the assistant into an elaborate dance: unbind GPUs from the NVIDIA driver, remove them from the PCI bus, perform Secondary Bus Reset (SBR) on the upstream bridges to clear GPU firmware state, rescan the bus, set identity domains before any driver binds, and then load NVIDIA. The preceding messages (6288–6299) show this dance in painful detail — SBRs that don't fully clear the GSP (GPU System Processor) firmware, GPUs that get grabbed by vfio-pci or nvidia at the wrong moment, and the discovery that Blackwell's FSP firmware survives software-initiated resets.

The Assumptions Behind the Attempt

Message <msg id=6300> represents a specific hypothesis: that if we can get the GPUs into a clean state (no driver bound, nvidia unloaded, SBR performed), set identity domains, and then load nvidia, the GPUs will initialize correctly with identity IOMMU and P2P will work.

This hypothesis rested on several assumptions:

The SBR fully resets the GPU firmware. The assistant had previously observed that SBR worked on the very first attempt (when GPUs came from a clean power-on state), but subsequent attempts failed. The assumption was that the sequence of remove → SBR → rescan (without any driver grabbing the device) would produce a clean state equivalent to power-on.
The IOMMU group type is a runtime-configurable property. The kernel exposes /sys/kernel/iommu_groups/$group/type as a writable file, and the assistant had successfully changed it before. The assumption was that this setting would be honored by the NVIDIA driver during device initialization.
The NVIDIA driver can initialize GPUs in identity IOMMU mode. This is the critical assumption — that the driver's initialization sequence (RmInitAdapter, GSP firmware boot, etc.) is agnostic to the IOMMU translation mode.
The GPUs are in a recoverable state. Despite the earlier "bad register read" errors (0xbadf4100) and "RmInitAdapter failed" messages, the assistant assumed that the remove/SBR/rescan cycle had cleared the corrupted firmware state. None of these assumptions turned out to be fully correct.

What Actually Happened: The Blackwell FSP Revelation

The "No devices were found" output is the symptom. The root cause, discovered in the following messages, is far more fundamental. The Blackwell GPU's Firmware Security Processor (FSP) — a dedicated security co-processor that manages GPU initialization, firmware validation, and secure boot — requires specific DMA mappings set up by the kernel's DMA API during its boot sequence. When the IOMMU is in identity mode, the kernel's DMA API does not create the translation mappings that the FSP expects. The FSP boot fails with error code 0x177, and the GPU never reaches a state where the NVIDIA driver can communicate with it.

The error manifests as RmInitAdapter failed! (0x62:0xffff:2142) — a generic initialization failure that traces back to the FSP's inability to complete its boot sequence. The 0xbadf4100 register reads (a known "bad register" signature on GH100/GB100 architectures) indicate that the GPU's internal control registers are returning garbage because the firmware never initialized them.

This is not a recoverable state through software means. The FSP firmware lives in protected on-chip SRAM that survives SBR, FLR (Function Level Reset), and even CXL bus resets. The only way to clear it is a full power cycle (D3cold → D0 transition) or physically powering the system off and on. The assistant's subsequent attempts to use nvidia_gpu_tools.py with --reset-with-sbr and --set-next-sbr-to-fundamental-reset all failed because fundamental reset is not supported on these GPUs, and SBR cannot touch the FSP's secure enclave.

The Deeper Technical Insight

The Blackwell FSP represents a significant architectural change from previous GPU generations. On Ampere (GH100) and earlier GPUs, the GSP (GPU System Processor) firmware could be reset via SBR because it loaded from VRAM into a volatile region. On Blackwell, the FSP is a hardened security processor with its own dedicated firmware storage, protected by secure boot chains and cryptographic validation. This is part of NVIDIA's response to increasing concerns about GPU firmware security — but it has the side effect of making the GPU much harder to recover from certain error states without a physical power cycle.

The IOMMU identity mode incompatibility is particularly insidious because it's not a bug — it's a feature collision. The FSP's secure boot sequence uses DMA to validate firmware signatures, check hardware integrity, and establish trust anchors. These DMA transactions require the IOMMU to provide specific translation services that identity mode bypasses. In translation mode, the kernel's DMA API creates IOMMU mappings that the FSP can use. In identity mode, those mappings don't exist, and the FSP's DMA transactions go directly to physical addresses — which the FSP may not be able to access without the IOMMU's address validation.

The Broader Impact

This message marks the end of one optimization path and the beginning of another. The assistant had been pursuing IOMMU identity domains as the primary solution for P2P DMA restoration. After this message, that path is definitively closed. The remaining options are:

Accept broken P2P and optimize around it. This means using NCCL_P2P_DISABLE=1 and relying on NVLink (if available) or host-mediated communication. The assistant had already set this flag.
Fix the NVIDIA driver's DmaRemapPeerMmio=1 parameter. This driver parameter is supposed to create proper IOMMU mappings for P2P access, but it was producing incomplete mappings (some peer pairs working, others faulting). This remains an open investigation.
Optimize MTP (Multi-Token Prediction) speculation. The assistant had already enabled MTP/NEXTN speculation, which provides 12-45% per-request throughput improvement at low concurrency. With P2P off the table, speculation becomes the primary optimization lever. The system ultimately stabilizes with a working configuration: 4 GPUs on the NVIDIA driver with DMA-FQ IOMMU type, P2P disabled, MTP speculation enabled, and SGLang serving the Qwen3.5-122B model at competitive throughput. The 4 VFIO GPUs remain on vfio-pci for the SEV-SNP VM, unaffected by the IOMMU experiments.

Conclusion

Message <msg id=6300> is a study in the limits of software-level system optimization. The assistant executed a technically correct sequence of operations — unbind, remove, SBR, rescan, set identity, load driver — and every step reported success. Yet the final result was failure, because the Blackwell GPU's firmware security architecture places constraints that no amount of PCI bus manipulation can overcome.

The lesson is both specific and general. Specifically, IOMMU identity domains are incompatible with Blackwell GPUs due to FSP boot requirements. Generally, modern GPU architectures are increasingly hardened against the kinds of low-level system manipulations that experienced engineers rely on for performance tuning. The Blackwell FSP, secure boot chains, and protected firmware regions represent a new class of constraints that system architects must account for.

For the assistant, this message represents the pivot point — the moment when a promising optimization path is abandoned and the focus shifts to working within the hardware's actual constraints. The system that emerges is stable, well-understood, and performing well, even if it doesn't achieve the P2P throughput that was the original goal. Sometimes the most valuable outcome of an engineering investigation is not the solution you were looking for, but the clear understanding of why it cannot work.