The Moment of Failure: When IOMMU Identity Domains Meet Blackwell's GSP Firmware

In the long arc of deploying and optimizing large language model inference on a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a message that marks a decisive turning point — a moment when a carefully researched plan collides with an undocumented hardware reality. Message 6265 is that moment. It is the instant when the assistant, having successfully unbounded four GPUs from the nvidia driver and switched their IOMMU group types from DMA-FQ to identity, attempts to rebind them and discovers that the Blackwell architecture has a hidden dependency that renders the entire approach unworkable.

The Context: A Quest for P2P DMA

To understand why message 6265 matters, we must first understand the problem it was trying to solve. The system in question is a Proxmox host running Ubuntu 24.04 with eight NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs on NUMA0, used for SGLang serving the Qwen3.5-122B-A10B BF16 model) and a SEV-SNP confidential VM (4 GPUs on NUMA1, passed through via vfio-pci). The host has amd_iommu=on and mem_encrypt=on (SEV-SNP) in its kernel command line, which means the AMD IOMMU is in full translation mode — every DMA operation from the GPUs must go through IOMMU page tables.

This is a problem for GPU-to-GPU communication. NVIDIA's NCCL library, which handles the all-reduce operations required for tensor-parallel inference, normally uses P2P (peer-to-peer) DMA to transfer data directly between GPU memory spaces. But with the IOMMU in full translation mode, P2P DMA breaks — the IOMMU cannot translate the peer addresses correctly, resulting in IO_PAGE_FAULT events logged by the AMD-Vi driver. The symptom is that NCCL falls back to using shared memory (SHM) copies through the CPU, which is 5–10× slower than direct GPU P2P transfers. For a MoE model like Qwen3.5-122B-A10B running with tensor parallelism across 4 GPUs, every all-reduce operation is slowed by this bottleneck.

The solution that the assistant had researched extensively in the preceding messages (see [msg 6247] through [msg 6264]) was to use a Linux kernel feature introduced in v5.11: per-IOMMU-group domain type switching. By writing "identity" to /sys/kernel/iommu_groups/<group>/type, individual IOMMU groups can be switched from translation mode (DMA-FQ) to identity (passthrough) mode, effectively disabling IOMMU translation for just those devices. The plan was to switch the four nvidia GPU groups to identity mode while leaving the four vfio-pci GPU groups in translation mode for SEV-SNP. This would restore P2P DMA for the inference GPUs without compromising the confidential VM's security guarantees.

The Message: Rebinding After Identity Mode

Message 6265 is the execution of the rebinding step. The assistant had already completed the unbinding and identity-setting in [msg 6263], and now it issues the command to bind the GPUs back to the nvidia driver:

ssh root@10.1.2.6 'set -e
GPUS="0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0"

echo "=== Rebinding GPUs to nvidia ==="
for gpu in $GPUS; do
    echo "$gpu" > /sys/bus/pci/drivers/nvidia/bind 2>/dev/null && echo "  $gpu bound to nvidia OK" || echo "  $gpu: FAILED to bind"
done
sleep 1

echo ""
echo "=== Restarting nvidia-persistenced ==="
systemctl start nvidia-persistenced
sleep 1

echo ""
echo "=== Final verification ==="
for gpu in $GPUS; do
    group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group))
    driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none")
    type=$(cat /sys/kernel/iommu_groups/$group/type)
    echo "GPU $gpu -> group=$group, driver=$driver, type=$type"
done

echo ""
echo "=== nvidia-smi ==="
nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader'

The output at first seems encouraging:

=== Rebinding GPUs to nvidia ===
  0000:01:00.0 bound to nvidia OK
  0000:11:00.0 bound to nvidia OK
  0000:61:00.0 bound to nvidia OK
  0000:71:00.0 bound to nvidia OK

=== Restarting nvidia-persistenced ===
Failed to start nvidia-persistenced.service: Unit nvidia-persistenced.service not found.

All four GPUs bind successfully. The only hiccup is that nvidia-persistenced.service is not found — but this is a Proxmox host, not a standard Ubuntu installation, and the service may be managed differently or not needed. The assistant does not yet know that something far more serious has gone wrong.## What the Output Doesn't Show: The Hidden Failure

The message's output ends abruptly. The final verification section — the nvidia-smi call — is missing from the output shown. The conversation data shows that the command was issued, but the output cuts off after the nvidia-persistenced failure message. This is because the bash tool terminated after exceeding a 30-second timeout (as revealed in the subsequent message, [msg 6266]). The nvidia-smi command hung because the GPUs were in a catastrophic state.

The next message ([msg 6266]) reveals the truth:

No devices were found

The GPUs bound successfully to the nvidia driver at the PCI level, but the driver's initialization of the GPU firmware failed. The kernel log entries (visible in [msg 6267] and [msg 6270]) tell the story:

NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x40:2142)
NVRM: GPU0 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
NVRM: GPU0 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)

The critical error is _kgspBootGspRm: unexpected WPR2 already up. WPR2 stands for "Write Protected Region 2" — a protected memory region used by the GSP (GPU System Processor, the Blackwell generation's firmware security processor). When the GPUs were unbounded from the nvidia driver, the GSP firmware was left in a state where WPR2 remained locked from the previous session. When the driver tries to reinitialize the GPU, it expects WPR2 to be in a clean state, but finds it already active and refuses to proceed.

This is the moment of discovery: IOMMU identity domains are fundamentally incompatible with Blackwell GPUs when set dynamically. The sequence of unbinding the driver, changing the IOMMU domain type, and rebinding the driver causes the GSP firmware to lose its initialization context. The firmware cannot be reinitialized without a full hardware reset that clears the WPR2 state.

The Assumptions That Failed

The assistant made several reasonable assumptions that turned out to be incorrect:

That unbinding and rebinding the nvidia driver is a safe operation. On previous GPU generations (Ampere, Hopper), unbinding and rebinding the driver works cleanly — the GPU firmware resets, the driver reinitializes everything, and the GPU comes back online. Blackwell's GSP firmware is more stateful and does not tolerate losing its driver context.
That PCI remove/rescan would fully reset the GPU state. When the GPUs failed to initialize, the assistant attempted a PCI remove followed by a rescan ([msg 6268]). This re-enumerated the devices and even re-probed the nvidia driver, but the GSP firmware state persisted. The unexpected WPR2 already up error appeared again.
That FLR (Function Level Reset) would clear the GSP state. The GPUs support FLR (the reset_method file showed flr bus cxl_bus), and the FLR operation reported success ([msg 6271]). But FLR only resets the PCI function — it does not reset the GSP firmware's WPR2 state. The firmware survived the FLR intact.
That SBR (Secondary Bus Reset) would be sufficient. Even a secondary bus reset via the parent PCIe bridge ([msg 6275]) failed to clear the GSP state. The assistant exhausted every software-level reset mechanism available: driver unbind/rebind, PCI remove/rescan, FLR, and SBR. None worked.

The Deeper Lesson: Blackwell's GSP Dependency

The root cause is architectural. NVIDIA's Blackwell GPUs introduced a more sophisticated firmware security processor (the GSP) that manages GPU initialization, power management, and secure boot. The GSP requires specific DMA mappings set up by the kernel's DMA API during initialization — mappings that only exist when the IOMMU is in translation mode. When the IOMMU group is switched to identity mode before the driver binds, the kernel's DMA API does not create the necessary translation mappings for the GSP's WPR2 region. The GSP firmware then fails to boot, and the GPU becomes permanently unresponsive until a cold reboot of the entire system.

This is not a bug that can be fixed by changing the timing or sequence of operations. The assistant tried setting identity mode before binding ([msg 6272]), after binding, and while the GPUs were in various reset states. The result was always the same: RmInitAdapter failed! (0x62:0x40:2142). The only way to recover the GPUs was a full system reboot, which the assistant eventually performed (as documented in the segment summary).

Input Knowledge Required

To fully understand message 6265, one needs:

Understanding of IOMMU concepts: What DMA-FQ vs identity domain types mean, how per-group domain switching works in Linux, and why identity mode would benefit GPU P2P DMA.
Knowledge of NVIDIA GPU architecture: The difference between the GPU's PCI function (which can be reset via FLR) and the GSP firmware (which maintains persistent state across resets).
Familiarity with Linux PCI subsystem: How driver unbind/bind works via sysfs, what PCI remove/rescan does, and the hierarchy of PCI reset methods (FLR, bus reset, SBR).
Context of the broader deployment: The SEV-SNP VM requiring amd_iommu=on, the split of 8 GPUs between LXC and VFIO, and the NCCL P2P disable workaround that was in place.

Output Knowledge Created

Message 6265 produced critical negative knowledge — knowledge of what does not work:

Blackwell GPUs cannot tolerate IOMMU domain type changes after driver initialization. The GSP firmware's WPR2 state is tied to DMA mappings that only exist in translation mode.
No software-level reset can clear this state. FLR, SBR, PCI remove/rescan, and driver unbind/rebind all fail. Only a full system power cycle (ACPI G3 -> S0) can reset the GSP firmware.
Per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs on this platform. The approach that works on paper (and on older GPU generations) hits a hardware/firmware limitation specific to Blackwell.
The DmaRemapPeerMmio=1 nvidia driver parameter remains as the only potential avenue for P2P restoration, though it was already enabled and produced incomplete IOMMU mappings (some peer pairs working, others faulting).

The Thinking Process Visible in the Message

Message 6265 itself is a straightforward execution command — it does not contain explicit reasoning. But the reasoning is visible in its structure:

The command is carefully sequenced: bind GPUs first, then restart nvidia-persistenced, then verify. This shows the assistant's mental model of the dependency chain.
The set -e at the top indicates the assistant expects all steps to succeed and wants the script to abort on any failure.
The verification section checks driver binding, IOMMU group type, and nvidia-smi output — a thorough validation that would catch any partial failures.
The nvidia-persistenced restart is a defensive measure to ensure the GPUs stay initialized, even though it's not strictly required for the identity domain change. The absence of the nvidia-smi output in the message is itself informative — it tells us the command timed out, which is the first symptom of the GPU initialization failure. The assistant would not learn the full extent of the problem until the next message, but the seed of the discovery is already planted in message 6265's incomplete output.

Conclusion

Message 6265 is a study in how the best-laid plans can collide with undocumented hardware behavior. The assistant had done thorough research, verified that the kernel supported per-group identity domains, confirmed that each GPU was alone in its IOMMU group, and planned a careful sequence of operations. Yet the Blackwell GSP firmware's hidden dependency on IOMMU translation mode rendered the entire approach invalid. The message marks the boundary between what is theoretically possible (per-group IOMMU identity domains) and what is practically achievable (stable GPU operation with IOMMU in translation mode). It is a reminder that in systems engineering, the most important discoveries are often the ones that tell you what cannot be done.