The Blackwell FSP Barrier: A Definitive End to the IOMMU Identity Domain Path

Introduction

In the relentless pursuit of GPU inference performance, few bottlenecks are as frustrating as the ones imposed by security infrastructure. This article chronicles the final chapter of a multi-session effort to restore GPU peer-to-peer (P2P) DMA on a system of eight NVIDIA RTX PRO 6000 Blackwell GPUs operating under full IOMMU translation mode — an effort that culminated in the definitive discovery that the Blackwell Firmware Security Processor (FSP) is fundamentally incompatible with IOMMU identity domains, closing off an entire line of optimization and forcing a graceful pivot to alternative performance strategies.

The segment spans roughly 180 messages across a single opencode coding session, tracing an arc from careful planning through methodical execution, repeated failure, creative workarounds, and ultimately the hard-won conclusion that per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs. Along the way, it illuminates the intricate dance between Linux kernel subsystems, NVIDIA's proprietary driver stack, PCIe reset mechanisms, and the immutable physics of firmware initialization. The story is one of disciplined engineering — of forming hypotheses, testing them rigorously, documenting failures, and knowing when to stop pushing against an immovable object.

The System and the Problem

The deployment in question is a Proxmox host (kpro6) running Ubuntu 24.04 with eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, each with 96 GB of GDDR7 memory, connected via PCIe Gen5 x16. The machine is split 4+4 across two NUMA domains: four GPUs on NUMA 0 are bound to the nvidia driver and used by an LXC container running SGLang for LLM inference with the Qwen3.5-122B-A10B BF16 model, while the four GPUs on NUMA 1 are bound to vfio-pci for passthrough to a SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) confidential VM managed by another tenant.

The critical constraint is that SEV-SNP requires the host's IOMMU to operate in full translation mode (amd_iommu=on without iommu=pt). This configuration, while essential for the confidential VM's security guarantees, creates a devastating side effect: GPU-to-GPU P2P DMA breaks completely. When one GPU attempts to directly write to another GPU's memory via P2P DMA, the IOMMU has no mapping for the target GPU's BAR (Base Address Register) region, causing IO_PAGE_FAULT errors and silent data corruption. The symptom is insidious: torch.cuda.can_device_access_peer() returns True (the driver believes P2P is functional), but actual data transfers produce mismatches on every GPU pair.

The workaround — disabling NCCL P2P entirely with NCCL_P2P_DISABLE=1 and falling back to shared memory transport — is functional but suboptimal. For a tensor-parallel MoE model like Qwen3.5-122B-A10B BF16, the all-reduce overhead from system memory copies represents a significant performance penalty. The nvidia driver's DmaRemapPeerMmio=1 parameter was already enabled but produced incomplete IOMMU mappings — some peer pairs worked while others continued to fault. The search for a better solution led to the most promising lead: per-group IOMMU identity domains.

The Promising Lead: Per-Group IOMMU Identity Domains

The user's explicit question in [msg 6238] — "Is there anything that can be done to get p2p going with iommu?" — set the assistant on a path of investigation that led to a powerful kernel feature. Since Linux kernel v5.11, the sysfs interface at /sys/kernel/iommu_groups/<grp_id>/type allows per-group switching between translation (DMA-FQ) and passthrough (identity) modes. This means the four nvidia GPU groups could be set to identity mode (effectively passthrough for those devices only) while keeping the four vfio-pci groups in DMA-FQ translation mode for SEV-SNP.

The assistant dispatched a research subagent in [msg 6247] to investigate the feasibility of this approach. The research confirmed that the kernel (6.14.11) fully supported the feature, each GPU was alone in its IOMMU group (ACS was enabled, providing clean isolation), and the VFIO groups were completely independent. The plan was methodical: stop SGLang, unbind the nvidia driver from all four GPUs, write identity to each group's type file, rebind the nvidia driver, verify with a P2P transfer test, and if successful, make the configuration persistent.

The user's response in [msg 6246] — "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed" — was a delegation signal, handing the initiative to the assistant. The assistant prioritized P2P restoration first, since the user had specifically asked about it.

The Reconnaissance Before the Cut

Before any changes were made, the assistant performed a thorough reconnaissance in [msg 6248]. Three bash commands probed different layers of the system:

The IOMMU group map: Iterating over all eight GPU PCI addresses to resolve IOMMU group numbers, bound drivers, and domain types. The output revealed the complete mapping: groups 42, 61, 28, 10 for the nvidia GPUs on NUMA 0, and groups 101, 117, 90, 72 for the vfio-pci GPUs on NUMA 1. All were in DMA-FQ mode.
Kernel version and boot parameters: The kernel was 6.14.11 (supporting the feature since 5.11), and the boot cmdline confirmed amd_iommu=on, mem_encrypt=on, and kvm_amd.sev=1 — all required for SEV-SNP and none of which would be changed.
IOMMU fault history: The dmesg output showed the familiar IO_PAGE_FAULT errors with GPU BAR addresses, confirming the P2P DMA problem was active and providing a baseline for verification. This reconnaissance was the pre-flight check — the careful measurement that precedes any surgical intervention on a running production system. Every assumption was validated against the current state before any change was made.

The Execution and Immediate Failure

The assistant proceeded with the plan in [msg 6256]. SGLang was stopped, zombie processes were cleaned up, and the four GPUs were unbounded from the nvidia driver. The identity domains were successfully written to each IOMMU group's type file. The GPUs were then rebounded to the nvidia driver.

Then came the verification in [msg 6266]:

nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader
No devices were found

This single line of output signaled the collapse of the carefully planned optimization. The nvidia driver had loaded successfully, but it could not initialize any of the four GPUs. The dmesg logs in [msg 6267] revealed the root cause: RmInitAdapter failed! (0x62:0xffff:2142) followed by kfspSendBootCommands_HAL errors and the ominous 0xbadf4100 register reads. The Blackwell GPU's Firmware Security Processor (FSP) had entered a locked state.

The Desperate Catalog of Software Resets

What followed was a systematic attempt to recover the GPUs using every available PCIe software reset mechanism. The assistant's debugging odyssey through [msg 6268] to [msg 6314] tested:

Function Level Reset (FLR): The standard PCIe reset that resets the device's PCI configuration space and internal state. On Blackwell, FLR completed without error but did not clear the FSP — the firmware remained in its corrupted state.

Secondary Bus Reset (SBR): A more aggressive reset that asserts the PCIe reset signal on a downstream port, resetting all devices behind a bridge. The assistant performed SBR on the parent bridges with varying hold times (200 ms, 500 ms). SBR also failed to clear the FSP.

CXL Bus Reset: A new reset type available on CXL-capable devices. The GPUs supported cxl_bus reset and the sysfs write returned "OK," but the subsequent nvidia driver load still failed with the same FSP initialization error.

NVIDIA's gpu-admin-tools: The assistant tried --reset-with-sbr and --set-next-sbr-to-fundamental-reset options. The fundamental reset was not supported, and the SBR option produced the same results as manual SBR.

D3cold power state: The assistant investigated whether transitioning the GPUs through the deepest PCIe power state (D3cold, where power is fully removed) could clear the FSP. The d3cold_allowed flag was set, but the power state was "unknown" — the GPU could not even participate in power state negotiation.

The error signature was consistent across all attempts: RmInitAdapter failed! (0x62:0xffff:2142) followed by kfspSendBootCommands_HAL errors and the cryptic gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110094, regvalue: 0xbadf4100. The error code 0x62 indicated a GSP (GPU System Processor) firmware initialization failure, and the 0xbadf4100 register value suggested the FSP was reading back garbage — a classic sign of a device put into an inconsistent state that no software reset could recover.

The One Sequence That Worked (But Not Enough)

Through this catalog of failures, one sequence consistently succeeded in bringing the GPUs back: PCI remove → SBR → rescan. When the GPUs were removed from the PCI bus, the parent bridges were reset via SBR, and the bus was rescanned, the GPUs reappeared with clean PCI configuration space and the nvidia driver initialized them successfully.

However, there was a critical catch: during the PCI rescan, the nvidia driver auto-probed the freshly discovered GPUs before the assistant could set the IOMMU groups to identity mode. The groups reverted to DMA-FQ — the default type — because the IOMMU group objects were destroyed when the devices were removed and recreated with default settings when they reappeared.

This timing constraint became the central challenge. In [msg 6278], the assistant confirmed the pattern:

0000:01:00.0: group=28, type=DMA-FQ
0000:11:00.0: group=61, type=DMA-FQ
0000:61:00.0: group=42, type=DMA-FQ
0000:71:00.0: group=10, type=DMA-FQ

The GPUs were back, all four working, but the identity domains were lost. The assistant had a working recovery procedure but no way to apply the identity configuration before the driver claimed the devices.

The Clever Workarounds and Their Failure

The assistant explored increasingly creative approaches to solve the timing problem:

The modprobe blacklist approach ([msg 6308] to [msg 6315]): By creating a temporary modprobe blacklist with install=/bin/false directives, the assistant prevented the nvidia driver from auto-probing during PCI rescan. The sequence was: unload nvidia modules, create blacklist, remove GPUs from PCI, SBR the bridges, rescan (GPUs appear driverless because nvidia is blacklisted), set identity domains, remove blacklist, manually load nvidia modules. This approach almost worked — the IOMMU groups were successfully set to identity mode before the driver loaded. But when the nvidia modules were loaded, the result was the same: "No devices were found."

The udev rule approach ([msg 6305] to [msg 6307]): The assistant considered creating a udev rule that would fire when the GPU device appeared on PCI, setting driver_override to prevent nvidia from claiming it, then setting identity and manually binding. However, this approach faced a race condition: the udev RUN script executes after the device is added, which may be after the driver has already bound.

The driver_override approach ([msg 6279]): The assistant attempted to use driver_override to prevent auto-probing, but discovered that the sysfs entry doesn't survive a PCI remove/rescan cycle, making it impractical for this use case.

The single-GPU test ([msg 6281]): In a moment of disciplined debugging, the assistant tested with just one GPU to minimize blast radius. The test was clean: unbind GPU 0000:01:00.0, set its IOMMU group to identity, rebind. The result was definitive: the GPU disappeared from nvidia-smi, leaving only three GPUs visible. The GSP was freshly reset via SBR, yet identity mode still caused the GPU to fail.

This single-GPU test was the final nail. It proved that the problem was not a stale GSP state from a previous unbind/rebind cycle, nor a timing issue with auto-probing, nor a race condition with udev. The problem was fundamental: Blackwell's FSP requires DMA translation mode during its initialization sequence.

The Modprobe Install Hook: A Boot-Time Solution

The insight that emerged from these runtime failures was that the identity domains had to be set before the nvidia driver ever touched the GPUs — during the very first module load at boot time. The assistant engineered an elegant solution: a modprobe install hook at /etc/modprobe.d/nvidia-iommu-identity.conf that would intercept the modprobe nvidia call triggered by PCI modalias matching, set the IOMMU groups to identity mode, and then load the real nvidia module via exec /sbin/modprobe --ignore-install nvidia "$@". This ensured the IOMMU type was already identity when nvidia's probe function ran for the first time on a fresh-from-power-on GPU.

The hook script, gpu-set-identity-before-nvidia.sh, was deployed to /usr/local/bin/ and tested at runtime. It worked correctly — the identity domains were set, and the journalctl logs confirmed each transition: "Set group 42 to identity for 0000:01:00.0 (was DMA-FQ)." The old systemd-based service (gpu-iommu-identity.service) was disabled in favor of this more precise mechanism.

The user executed the reboot in [msg 6410], and the assistant waited for the host to come back up.

The Reboot and the Discovery of Error 0x177

The first check in [msg 6414] showed a stark result: nvidia-smi returned "No devices were found." This was deeply confusing because the nvidia kernel modules were loaded and all four GPUs showed driver=nvidia in sysfs.

The assistant's diagnostic response in [msg 6415] is a masterclass in systematic debugging. The assistant ran a carefully structured command that checked four hypotheses in sequence:

Is the nvidia module loaded? lsmod | grep nvidia confirmed all four submodules (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm) were loaded with normal reference counts.
Are the GPUs bound to the nvidia driver? The loop over the four PCI addresses confirmed each showed driver=nvidia.
Did the modprobe hook set identity domains? All four IOMMU groups showed type=identity. The hook worked perfectly.
Are there kernel errors? The dmesg grep for NVRM|gpu-iommu|nvidia and the journalctl query for the hook's logger tag were designed to surface any NVIDIA kernel module errors. The dmesg output, fully visible in [msg 6418], contained the bombshell:

NVRM: GPU0 kfspProcessCommandResponse_GH100: FSP response reported error.
Task ID: 0x1 Command type: 0x14 Error code: 0x177
NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read:
addr: 0x110040, regvalue: 0xbadf4100

The error code 0x177 was different from the 0xffff errors seen during the earlier runtime unbind attempts. This was happening on a fresh boot — the GPUs had clean FSP state from power-on. The modprobe hook had set identity domains before nvidia loaded, just as designed. And yet the FSP boot failed.

This led to the inescapable conclusion: IOMMU identity mode itself breaks the Blackwell FSP boot process. The FSP firmware requires specific DMA mappings that are set up by the kernel's DMA API in translation mode. In identity mode, the IOMMU is bypassed, so the kernel never creates the translation tables that the FSP expects. The FSP's boot command sequence fails because it cannot communicate properly with the GPU's hardware through the identity-mapped IOMMU.

This is a hardware/firmware limitation, not a software bug. The Blackwell GPU's FSP was designed to operate with an active IOMMU that provides DMA translation. Identity mode removes that translation, and the FSP's initialization path doesn't handle it. The 0xbadf4100 register value is a classic NVIDIA "poison" pattern — the FSP mailbox register (NV_PGSP_FALCON_MAILBOX0) containing this value means the firmware has entered a fault state from which it cannot recover within this boot session.

The Immediate Revert

The assistant's response in [msg 6419] was swift and decisive. There was no attempt to debug further, no experimentation with alternative configurations. The conclusion was immediate: identity IOMMU mode breaks the Blackwell FSP boot process. The modprobe hook was removed and the system was rebooted:

ssh root@10.1.2.6 'rm -f /etc/modprobe.d/nvidia-iommu-identity.conf && echo "Modprobe hook removed" && reboot'

This is a critical moment of engineering judgment. The assistant had invested significant effort across multiple sessions to enable P2P DMA — diagnosing the IO_PAGE_FAULT errors, discovering the GSP stickiness, crafting the modprobe hook, analyzing boot ordering — and had to accept that the approach was fundamentally blocked by Blackwell's firmware architecture. The graceful retreat to a working configuration, with MTP speculation as the primary optimization, demonstrates the engineering maturity to know when to stop pushing against an immovable object and consolidate the gains already achieved.

After the revert reboot, the verification in [msg 6421] confirmed the system was whole again:

=== nvidia-smi ===
0, 00000000:01:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB
1, 00000000:11:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB
2, 00000000:61:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB
3, 00000000:71:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB

=== IOMMU types ===
  0000:01:00.0: driver=nvidia, group=42, type=DMA-FQ
  0000:11:00.0: driver=nvidia, group=61, type=DMA-FQ
  0000:61:00.0: driver=nvidia, group=28, type=DMA-FQ
  0000:71:00.0: driver=nvidia, group=10, type=DMA-FQ

Every GPU was present, every IOMMU group was back to DMA-FQ (the default translation mode), and the VFIO-bound GPUs for the SEV-SNP VM remained on vfio-pci. The container was listed as running. The system was stable.

The False Alarm: A Health Check That Lied

After the revert, the assistant started the LXC container and launched SGLang. Then it ran a standard health check loop — a bash one-liner that curled the /v1/models endpoint every 10 seconds and broke when it got a 200 HTTP status code. This loop ran for 30 iterations — 300 seconds — without ever seeing a 200 response.

The assistant's diagnostic pivot in [msg 6425] is a textbook example of the "investigate before acting" operational pattern. Rather than immediately restarting the service or escalating, the assistant checked systemctl is-active and journalctl to verify the server's actual state:

Mar 09 21:12:12 llm-two sglang-qwen[221]: [2026-03-09 21:12:12] Successfully reserved port 30000
Mar 09 21:12:12 llm-two sglang-qwen[221]: [2026-03-09 21:12:12] INFO: Started server process [221]
Mar 09 21:12:12 llm-two sglang-qwen[221]: [2026-03-09 21:12:12] INFO: Waiting for application startup.

The server had started successfully. The health check was the problem, not the server. In [msg 6426], the assistant verified with a direct curl-to-JSON query that extracted the model ID: qwen3.5-122b. The server was not just running — it was fully functional, serving model metadata on demand.

This is a classic debugging pitfall: trusting a monitoring signal without validating it. The assistant's realization represents the moment it broke out of this assumption loop and cross-referenced the health check against the server's own logs. The logs told the truth; the health check was lying. The exact cause of the false negative was never identified, but the lesson was clear: even the most carefully designed health checks can produce false negatives, making the operator's judgment the most critical tool.

MTP Speculation: The Optimization That Survived

Throughout this entire saga — the IOMMU identity domain experiment, the FSP failure, the revert, the two reboots — one optimization remained stable: MTP (Multi-Token Prediction) speculative decoding. The MTP/NEXTN speculation had been enabled in the previous session and survived the reboot cycle intact.

The assistant had researched the required SGLang flags in a subagent task ([msg 6348]), which explored the SGLang codebase to identify the correct command-line parameters for hybrid Mamba/attention models like Qwen3.5-122B-A10B. The research revealed that --speculative-algorithm NEXTN was needed, which SGLang internally converts to the EAGLE algorithm path. The MTP draft model is loaded automatically from the model's own checkpoint — consuming 1.91 GB per GPU rank.

The verification in [msg 6362] confirmed that MTP was working with exceptional metrics: accept len: 1.95 and accept rate: 0.97. An acceptance rate of 0.97 means 97% of the tokens predicted by the draft model are accepted by the target model — an exceptionally high figure. An acceptance length of 1.95 means nearly two tokens are generated per speculation step on average, roughly doubling the effective decode throughput for single requests.

The comprehensive benchmark in [msg 6365] quantified the tradeoff across concurrency levels from 1 to 128. At low concurrency (C=1 through C=16), MTP delivered 12–45% per-request throughput improvement. At C=64 and C=128, aggregate throughput dropped by 32–42% due to the max_running_requests=48 cap that SGLang automatically imposes when speculative decoding is enabled, combined with the extra memory overhead of the draft model's mamba state.

For the user's primary use case — low-concurrency agentic coding with long contexts — this tradeoff was a clear win. The per-request improvement in response latency matters more than aggregate throughput when individual users are waiting for responses.

The Empty Message and the Summary

The segment concludes with a remarkable moment: the user sends an empty message ([msg 6427]), consisting solely of XML wrapper tags with zero content. This absence of content is itself deeply meaningful. Within the conventions of this AI-assisted workflow, the empty message signals "continue" or "proceed" — the user trusts the assistant to summarize the findings and propose next steps without needing explicit instructions.

The assistant's response is one of the most information-dense messages in the entire session — a comprehensive document covering every discovery, every configuration detail, every benchmark result, and every file location. It transforms the session's implicit knowledge into explicit, structured, queryable documentation that serves as a handoff document, a system state snapshot, and a decision record all in one.

Key Outcomes and Knowledge Created

This segment produced several critical pieces of knowledge:

Definitive negative result: IOMMU identity domains are fundamentally incompatible with Blackwell GPUs. The FSP requires DMA translation mode during its boot sequence, and no software-level reset (FLR, SBR, CXL bus reset) can clear this state. This is a hardware/firmware limitation, not a software configuration issue.
Validated modprobe install hook pattern: Even though the approach ultimately failed, the hook mechanism itself was proven to work correctly. The identity domains were set before nvidia loaded, and the journalctl logs confirmed the transitions from DMA-FQ to identity. This pattern could be useful for other scenarios where pre-driver configuration is needed.
MTP speculation confirmed stable: The MTP/NEXTN speculation survived two rapid reboots and continues to deliver 12–45% per-request throughput improvement at low concurrency. The acceptance rate of 0.97 is exceptionally high, validating that the Qwen3.5 model's MTP head is well-calibrated.
Health check methodology improved: The false negative in the curl-based health check revealed that monitoring tools can be unreliable. The assistant learned to cross-reference health checks against server logs and direct API queries.
Production recovery procedure validated: The sequence of events established a reliable recovery path: detect incompatibility, revert configuration, reboot, verify GPUs, start container, start service. This is valuable operational knowledge for similar deployments.

Lessons for Blackwell GPU Deployments

This saga yields several important lessons for anyone deploying NVIDIA Blackwell GPUs in virtualized environments with IOMMU requirements:

1. Blackwell's FSP is fundamentally different from previous GPU architectures. On Ampere and Hopper GPUs, the GSP (GPU System Processor) could tolerate driver unbind/rebind cycles and IOMMU configuration changes. Blackwell's FSP introduces a new security architecture that is resistant to software-initiated resets and requires DMA translation mode during initialization.

2. Software resets cannot clear FSP state. FLR, SBR, and CXL bus reset all fail to clear the Blackwell FSP once it has been put into an inconsistent state. The only reliable recovery procedure is PCI remove → SBR → rescan, which produces a clean PCI configuration space but does not reset the FSP's fundamental dependency on DMA translation.

3. The PCI remove → SBR → rescan sequence is a reliable recovery procedure. While it cannot fix the identity domain incompatibility, this sequence successfully recovers Blackwell GPUs from GSP lockup states caused by other operations. It is a valuable tool for any Blackwell deployment.

4. Per-group IOMMU identity domains are not viable for Blackwell GPUs. The FSP's requirement for DMA translation during initialization makes identity mode fundamentally incompatible. Any deployment that needs both IOMMU protection (for security or virtualization) and GPU P2P DMA must pursue alternative approaches, such as the nvidia driver's DmaRemapPeerMmio parameter.

5. MTP speculation provides meaningful throughput improvement even without P2P. The 12–45% per-request boost from MTP speculation partially compensates for the loss of P2P DMA, and it survives reboots. It should be considered a primary optimization for Blackwell-based inference deployments.

Conclusion

The attempt to restore GPU P2P DMA via per-group IOMMU identity domains on NVIDIA Blackwell GPUs was a journey from careful planning through methodical execution, repeated failure, creative workarounds, and ultimately the hard-won discovery of a fundamental hardware constraint. The Blackwell FSP's requirement for DMA translation during initialization is an immovable barrier that no amount of software cleverness can overcome.

The assistant's systematic approach — reconnaissance before action, single-variable testing, comprehensive documentation of failures, and disciplined recovery — exemplifies the methodology required for debugging at the intersection of GPU firmware, IOMMU architecture, and Linux kernel subsystems. The negative result is as valuable as a success would have been: it closes off an entire line of inquiry, documents a critical hardware compatibility constraint, and forces a pragmatic focus on alternative optimization paths.

In the end, the system is stable. SGLang serves Qwen3.5-122B-A10B BF16 on four GPUs with TP=4, MTP speculation delivers 12–45% throughput improvement, and the deployment continues to function within the constraints imposed by the hardware. The P2P DMA question remains open — the nvidia driver's DmaRemapPeerMmio=1 parameter may yet yield results — but the identity domain approach has been definitively laid to rest. Sometimes the most important thing a debugger can learn is what doesn't work, and why.