The Blackwell FSP Barrier: How IOMMU Identity Domains Met Their Match in NVIDIA's Firmware Security Processor

Introduction

In the landscape of modern AI infrastructure, few challenges are as technically demanding as deploying multi-GPU LLM inference systems with stringent security requirements. This article chronicles a pivotal chapter in such a deployment: the systematic attempt to restore GPU peer-to-peer (P2P) DMA on a system of eight NVIDIA RTX PRO 6000 Blackwell GPUs operating under full IOMMU translation mode, and the discovery that a fundamental hardware constraint — the Blackwell Firmware Security Processor's (FSP) dependency on DMA translation during initialization — renders the most promising software-based approach definitively unworkable.

The story spans dozens of messages across a single segment of an opencode coding session, tracing an arc from careful planning through methodical execution, repeated failure, creative workarounds, and ultimately the hard-won conclusion that per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs. Along the way, it illuminates the intricate dance between Linux kernel subsystems, NVIDIA's proprietary driver stack, PCIe reset mechanisms, and the immutable physics of firmware initialization.

The System and the Problem

The deployment in question is a Proxmox host (kpro6) running Ubuntu 24.04 with eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, each with 96GB of GDDR7 memory, connected via PCIe Gen5 x16. The machine is split 4+4 across two NUMA domains: four GPUs on NUMA 0 are bound to the nvidia driver and used by an LXC container running SGLang for LLM inference, while the four GPUs on NUMA 1 are bound to vfio-pci for passthrough to a SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) confidential VM managed by another tenant.

The critical constraint is that SEV-SNP requires the host's IOMMU to operate in full translation mode (amd_iommu=on without iommu=pt). This configuration, while essential for the confidential VM's security guarantees, creates a devastating side effect: GPU-to-GPU P2P DMA breaks completely. When one GPU attempts to directly write to another GPU's memory via P2P DMA, the IOMMU has no mapping for the target GPU's BAR (Base Address Register) region, causing IO_PAGE_FAULTs and silent data corruption. The symptom is insidious: torch.cuda.can_device_access_peer() returns True (the driver believes P2P is functional), but actual data transfers produce mismatches on every GPU pair.

The workaround — disabling NCCL P2P entirely with NCCL_P2P_DISABLE=1 and falling back to shared memory transport — is functional but suboptimal. For a tensor-parallel MoE model like Qwen3.5-122B-A10B BF16, the all-reduce overhead from system memory copies represents a significant performance penalty.

The Promising Lead: Per-Group IOMMU Identity Domains

The user's explicit question in [msg 6238] — "Is there anything that can be done to get p2p going with iommu?" — set the assistant on a path of investigation that led to a promising kernel feature. Since Linux kernel v5.11, the sysfs interface at /sys/kernel/iommu_groups/<grp_id>/type allows per-group switching between translation (DMA-FQ) and passthrough (identity) modes. This means the four nvidia GPU groups could be set to identity mode (effectively passthrough for those devices only) while keeping the four vfio-pci groups in DMA-FQ translation mode for SEV-SNP.

The assistant dispatched a research subagent in [msg 6247] to investigate the feasibility of this approach. The research confirmed that the kernel (6.14.11) fully supported the feature, each GPU was alone in its IOMMU group (ACS was enabled, providing clean isolation), and the VFIO groups were completely independent. The plan was methodical: stop SGLang, unbind the nvidia driver from all four GPUs, write identity to each group's type file, rebind the nvidia driver, verify with a P2P transfer test, and if successful, make the configuration persistent.

The user's response in [msg 6246] — "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed" — was a delegation signal, handing the initiative to the assistant. The assistant prioritized P2P restoration first, since the user had specifically asked about it.

The Reconnaissance Before the Cut

Before any changes were made, the assistant performed a thorough reconnaissance in [msg 6248]. Three bash commands probed different layers of the system:

  1. The IOMMU group map: Iterating over all eight GPU PCI addresses to resolve IOMMU group numbers, bound drivers, and domain types. The output revealed the complete mapping: groups 42, 61, 28, 10 for the nvidia GPUs on NUMA 0, and groups 101, 117, 90, 72 for the vfio-pci GPUs on NUMA 1. All were in DMA-FQ mode.
  2. Kernel version and boot parameters: The kernel was 6.14.11 (supporting the feature since 5.11), and the boot cmdline confirmed amd_iommu=on, mem_encrypt=on, and kvm_amd.sev=1 — all required for SEV-SNP and none of which would be changed.
  3. IOMMU fault history: The dmesg output showed the familiar IO_PAGE_FAULTs with GPU BAR addresses, confirming the P2P DMA problem was active and providing a baseline for verification. This reconnaissance was the pre-flight check — the careful measurement that precedes any surgical intervention on a running production system. Every assumption was validated against the current state before any change was made.

The Execution and Immediate Failure

The assistant proceeded with the plan in [msg 6256]-[msg 6265]. SGLang was stopped, zombie processes were cleaned up, and the four GPUs were unbounded from the nvidia driver. The identity domains were successfully written to each IOMMU group's type file. The GPUs were then rebounded to the nvidia driver.

Then came the verification in [msg 6266]:

nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader
No devices were found

This single line of output signaled the collapse of the carefully planned optimization. The nvidia driver had loaded successfully, but it could not initialize any of the four GPUs. The dmesg logs in [msg 6267] revealed the root cause: RmInitAdapter failed! (0x62:0xffff:2142) followed by kfspSendBootCommands_HAL errors and the ominous 0xbadf4100 register reads. The Blackwell GPU's Firmware Security Processor (FSP) had entered a locked state.

The Desperate Catalog of Software Resets

What followed was a systematic attempt to recover the GPUs using every available PCIe software reset mechanism. The assistant's debugging odyssey through [msg 6268]-[msg 6314] tested:

Function Level Reset (FLR): The standard PCIe reset that resets the device's PCI configuration space and internal state. On Blackwell, FLR completed without error but did not clear the FSP — the firmware remained in its corrupted state.

Secondary Bus Reset (SBR): A more aggressive reset that asserts the PCIe reset signal on a downstream port, resetting all devices behind a bridge. The assistant performed SBR on the parent bridges with varying hold times (200ms, 500ms). SBR also failed to clear the FSP.

CXL Bus Reset: A new reset type available on CXL-capable devices. The GPUs supported cxl_bus reset and the sysfs write returned "OK," but the subsequent nvidia driver load still failed with the same FSP initialization error.

NVIDIA's gpu-admin-tools: The assistant tried --reset-with-sbr and --set-next-sbr-to-fundamental-reset options. The fundamental reset was not supported, and the SBR option produced the same results as manual SBR.

D3cold power state: The assistant investigated whether transitioning the GPUs through the deepest PCIe power state (D3cold, where power is fully removed) could clear the FSP. The d3cold_allowed flag was set, but the power state was "unknown" — the GPU could not even participate in power state negotiation.

The error signature was consistent across all attempts: RmInitAdapter failed! (0x62:0xffff:2142) followed by kfspSendBootCommands_HAL errors and the cryptic gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110094, regvalue: 0xbadf4100. The error code 0x62 indicated a GSP (GPU System Processor) firmware initialization failure, and the 0xbadf4100 register value suggested the FSP was reading back garbage — a classic sign of a device put into an inconsistent state that no software reset could recover.

The One Sequence That Worked (But Not Enough)

Through this catalog of failures, one sequence consistently succeeded in bringing the GPUs back: PCI remove → SBR → rescan. When the GPUs were removed from the PCI bus, the parent bridges were reset via SBR, and the bus was rescanned, the GPUs reappeared with clean PCI configuration space and the nvidia driver initialized them successfully.

However, there was a critical catch: during the PCI rescan, the nvidia driver auto-probed the freshly discovered GPUs before the assistant could set the IOMMU groups to identity mode. The groups reverted to DMA-FQ — the default type — because the IOMMU group objects were destroyed when the devices were removed and recreated with default settings when they reappeared.

This timing constraint became the central challenge. In [msg 6278], the assistant confirmed the pattern:

0000:01:00.0: group=28, type=DMA-FQ
0000:11:00.0: group=61, type=DMA-FQ
0000:61:00.0: group=42, type=DMA-FQ
0000:71:00.0: group=10, type=DMA-FQ

The GPUs were back, all four working, but the identity domains were lost. The assistant had a working recovery procedure but no way to apply the identity configuration before the driver claimed the devices.

The Clever Workarounds and Their Failure

The assistant explored increasingly creative approaches to solve the timing problem:

The modprobe blacklist approach ([msg 6308]-[msg 6315]): By creating a temporary modprobe blacklist with install=/bin/false directives, the assistant prevented the nvidia driver from auto-probing during PCI rescan. The sequence was: unload nvidia modules, create blacklist, remove GPUs from PCI, SBR the bridges, rescan (GPUs appear driverless because nvidia is blacklisted), set identity domains, remove blacklist, manually load nvidia modules. This approach almost worked — the IOMMU groups were successfully set to identity mode before the driver loaded. But when the nvidia modules were loaded, the result was the same: "No devices were found."

The udev rule approach ([msg 6305]-[msg 6307]): The assistant considered creating a udev rule that would fire when the GPU device appeared on PCI, setting driver_override to prevent nvidia from claiming it, then setting identity and manually binding. However, this approach faced a race condition: the udev RUN script executes after the device is added, which may be after the driver has already bound.

The driver_override approach ([msg 6279]): The assistant attempted to use driver_override to prevent auto-probing, but discovered that the sysfs entry doesn't survive a PCI remove/rescan cycle, making it impractical for this use case.

The single-GPU test ([msg 6281]): In a moment of disciplined debugging, the assistant tested with just one GPU to minimize blast radius. The test was clean: unbind GPU 0000:01:00.0, set its IOMMU group to identity, rebind. The result was definitive: the GPU disappeared from nvidia-smi, leaving only three GPUs visible. The GSP was freshly reset via SBR, yet identity mode still caused the GPU to fail.

This single-GPU test was the final nail. It proved that the problem was not a stale GSP state from a previous unbind/rebind cycle, nor a timing issue with auto-probing, nor a race condition with udev. The problem was fundamental: Blackwell's FSP requires DMA translation mode during its initialization sequence.

The Root Cause: Error 0x177

The assistant eventually identified the root cause through careful analysis of the dmesg logs. The Blackwell FSP boot sequence fails with error code 0x177 when IOMMU is in identity mode. The FSP requires specific DMA mappings set up by the kernel's DMA API in translation mode during its initialization. When the IOMMU group is in identity mode, the kernel's DMA API does not create these mappings, and the FSP boot sequence fails before the nvidia driver even begins its initialization.

This is not a timing issue or a race condition — it is a fundamental hardware requirement. The FSP is a dedicated, security-hardened microcontroller embedded in the GPU that handles firmware validation, key management, and secure boot. On Blackwell GPUs, the FSP must perform DMA operations during its own initialization, before the nvidia driver even begins its rm_init_adapter sequence. If the IOMMU is in identity mode, the FSP's DMA accesses bypass translation entirely, but the kernel's DMA API has not set up the physical-to-device-address mappings that the FSP expects. The result is a boot failure that leaves the GPU in an unrecoverable state.

This discovery has profound implications. It means that per-group IOMMU identity domains are fundamentally incompatible with NVIDIA Blackwell GPUs — at least with the current driver and firmware. No amount of careful sequencing, udev rules, or boot-time configuration can work around a hardware-level dependency on DMA translation during firmware initialization.

The Reversion and Recovery

With the identity domain approach definitively blocked, the assistant immediately pivoted to recovery. The modprobe hook that had been installed to set identity domains at boot time was removed, and the system was rebooted. After the reboot, the GPUs returned to their working DMA-FQ configuration, and SGLang resumed serving the Qwen3.5-122B-A10B BF16 model with NCCL_P2P_DISABLE=1.

The MTP (Multi-Token Prediction) speculation that had been enabled in a previous session survived the reboot and continued to provide significant throughput improvements — 12-45% per-request boost at low concurrency. This became the primary active optimization, partially compensating for the loss of P2P DMA.

The Remaining Options

With the IOMMU identity domain path closed, the assistant identified the only remaining option for P2P DMA: the nvidia driver's DmaRemapPeerMmio=1 parameter. This parameter is designed to create special IOMMU mappings for P2P traffic, allowing peer-to-peer transfers even under IOMMU translation mode. It was already enabled on the system, but it produced incomplete results — some peer pairs worked while others continued to fault.

The assistant documented this as a potential avenue for future investigation, but the immediate priority was to stabilize the deployment with the working configuration: DMA-FQ IOMMU type, NCCL_P2P_DISABLE=1, and MTP speculation enabled. The system was stable, SGLang was serving requests, and the throughput improvement from MTP speculation provided a meaningful performance gain.

Lessons for Blackwell GPU Deployments

This saga yields several important lessons for anyone deploying NVIDIA Blackwell GPUs in virtualized environments with IOMMU requirements:

1. Blackwell's FSP is fundamentally different from previous GPU architectures. On Ampere and Hopper GPUs, the GSP (GPU System Processor) could tolerate driver unbind/rebind cycles and IOMMU configuration changes. Blackwell's FSP introduces a new security architecture that is resistant to software-initiated resets and requires DMA translation mode during initialization.

2. Software resets cannot clear FSP state. FLR, SBR, and CXL bus reset all fail to clear the Blackwell FSP once it has been put into an inconsistent state. The only reliable recovery procedure is PCI remove → SBR → rescan, which produces a clean PCI configuration space but does not reset the FSP's fundamental dependency on DMA translation.

3. The PCI remove → SBR → rescan sequence is a reliable recovery procedure. While it cannot fix the identity domain incompatibility, this sequence successfully recovers Blackwell GPUs from GSP lockup states caused by other operations. It is a valuable tool for any Blackwell deployment.

4. Per-group IOMMU identity domains are not viable for Blackwell GPUs. The FSP's requirement for DMA translation during initialization makes identity mode fundamentally incompatible. Any deployment that needs both IOMMU protection (for security or virtualization) and GPU P2P DMA must pursue alternative approaches, such as the nvidia driver's DmaRemapPeerMmio parameter.

5. MTP speculation provides meaningful throughput improvement even without P2P. The 12-45% per-request boost from MTP speculation partially compensates for the loss of P2P DMA, and it survives reboots. It should be considered a primary optimization for Blackwell-based inference deployments.

Conclusion

The attempt to restore GPU P2P DMA via per-group IOMMU identity domains on NVIDIA Blackwell GPUs was a journey from careful planning through methodical execution, repeated failure, creative workarounds, and ultimately the hard-won discovery of a fundamental hardware constraint. The Blackwell FSP's requirement for DMA translation during initialization is an immovable barrier that no amount of software cleverness can overcome.

The assistant's systematic approach — reconnaissance before action, single-variable testing, comprehensive documentation of failures, and disciplined recovery — exemplifies the methodology required for debugging at the intersection of GPU firmware, IOMMU architecture, and Linux kernel subsystems. The negative result is as valuable as a success would have been: it closes off an entire line of inquiry, documents a critical hardware compatibility constraint, and forces a pragmatic focus on alternative optimization paths.

In the end, the system is stable. SGLang serves Qwen3.5-122B-A10B BF16 on four GPUs with TP=4, MTP speculation delivers 12-45% throughput improvement, and the deployment continues to function within the constraints imposed by the hardware. The P2P DMA question remains open — the nvidia driver's DmaRemapPeerMmio=1 parameter may yet yield results — but the identity domain approach has been definitively laid to rest. Sometimes the most important thing a debugger can learn is what doesn't work, and why.