The Postmortem That Defined a Boundary: Why Blackwell GPUs Cannot Cross the IOMMU Translation Barrier

Introduction

In the sprawling ecosystem of high-performance AI inference infrastructure, few moments are as consequential as the one where a team realizes a fundamental hardware limitation cannot be worked around. The message at index 6428 of this opencode conversation is precisely such a document — a comprehensive technical postmortem, system state inventory, and architectural decision record rolled into one. Written by the AI assistant after an intensive session of trial, error, and eventual acceptance, this message represents the culmination of a deep investigation into whether GPU Peer-to-Peer (P2P) Direct Memory Access could be restored on a system where the IOMMU (Input/Output Memory Management Unit) was operating in full translation mode.

The answer, delivered with the weight of empirical evidence across multiple reboots, kernel configurations, and hardware resets, was a definitive no. And the reason — that identity IOMMU domains break the Blackwell GPU's Firmware Security Processor (FSP) boot sequence — is a finding that has implications far beyond this single Proxmox host.

This article examines message 6428 in depth: why it was written, what decisions it records, the assumptions it challenges, the knowledge it creates, and the thinking process that led to its conclusions. It is a case study in systematic hardware debugging, the value of thorough documentation, and the moment when engineering pragmatism must accept a hard constraint.

The Context: Eight Blackwell GPUs, Two Tenants, One IOMMU

To understand message 6428, one must first understand the infrastructure it describes. The system in question — a Proxmox host codenamed "kpro6" — is equipped with eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, each with 96GB of GDDR7 memory, connected via PCIe Gen5 x16. These are not consumer cards; they are professional-grade server GPUs designed for AI inference and training workloads, each costing thousands of dollars and representing a significant capital investment.

The machine is split 4+4: four GPUs on NUMA node 0 are bound to the nvidia driver and passed through to an LXC container running SGLang for LLM inference, while the other four GPUs on NUMA node 1 are bound to vfio-pci for passthrough to a SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) confidential VM managed by another tenant. This dual-tenant configuration is itself a complex engineering achievement — getting two independent GPU workloads to coexist on the same host with different security requirements and driver stacks is non-trivial.

The primary model deployed on the inference side is Qwen3.5-122B-A10B BF16 — a 125-billion-parameter Mixture-of-Experts model with only 10 billion active parameters per token, running at native BF16 precision (no quantization) across all four GPUs with tensor parallelism (TP=4). Multi-Token Prediction (MTP) speculation is enabled, providing a 12-45% per-request throughput improvement over the baseline.

This was the state of the system when the assistant began its investigation. The question on the table: could GPU P2P DMA be restored?

The Problem That Wouldn't Be Solved

P2P DMA is a critical capability for multi-GPU inference. When enabled, GPUs can directly read and write each other's memory across the PCIe fabric without involving the CPU or system memory as an intermediary. For tensor-parallel inference — where a single model's layers are sharded across multiple GPUs and every forward pass requires all-reduce operations to synchronize partial results — P2P DMA can dramatically reduce latency and improve throughput.

The problem was that P2P DMA was broken on this system. The root cause was the IOMMU configuration: the kernel was booted with amd_iommu=on (not iommu=pt, which would pass through DMA transactions without translation) because SEV-SNP required full IOMMU translation. In this mode, the IOMMU intercepts all DMA transactions, translating device addresses to physical addresses. For P2P DMA between GPUs, this translation layer introduces overhead and, more critically, can cause corruption when the IOMMU remaps peer memory addresses.

The nvidia driver had a parameter called DmaRemapPeerMmio=1 that attempted to work around this, but it only partially worked — transfers from lower-numbered GPUs to higher-numbered GPUs succeeded, while transfers in the reverse direction suffered corruption. The team had already fallen back to NCCL_P2P_DISABLE=1, forcing NCCL (the NVIDIA Collective Communications Library) to use SHM (shared memory) transport through the CPU instead of direct GPU-to-GPU transfers.

But this was a performance compromise, and the team wanted to know if there was a way to restore full P2P capability. The investigation that followed — documented in the session leading up to message 6428 — was thorough, methodical, and ultimately conclusive.

The Investigation: Three Approaches, Three Failures

The assistant's investigation into P2P restoration followed three distinct approaches, each of which was systematically tested and found to fail for fundamental reasons.

Approach 1: Per-IOMMU-Group Identity Domains

The first approach was the most promising. Modern Linux kernels (6.14 in this case) support per-IOMMU-group identity domains via the sysfs interface: writing identity to /sys/kernel/iommu_groups/XX/type tells the IOMMU to pass through DMA transactions for that group without translation. The idea was to set only the IOMMU groups containing the NUMA 0 GPUs to identity mode, leaving the NUMA 1 groups in translation mode for SEV-SNP.

The assistant confirmed that this mechanism worked at the kernel level — groups could be individually set to identity while others remained DMA-FQ (full translation with fine-grained queueing). A modprobe install hook was created at /etc/modprobe.d/nvidia-iommu-identity.conf that would intercept the modprobe nvidia call and set the identity domains before the nvidia driver ever touched the GPUs.

This was tested across two reboots. The first reboot revealed an ordering issue: the systemd gpu-vfio-split.service ran before the modprobe hook and grabbed all GPUs via new_id, preventing nvidia from binding. This was fixed. The second reboot was the critical test: the hook ran correctly, set all four NUMA 0 GPU groups to identity before nvidia loaded... and then nvidia's FSP boot failed with error code 0x177.

This was the moment of discovery. The Blackwell GPU's Firmware Security Processor (FSP) — a dedicated security coprocessor that handles firmware initialization, attestation, and protected region management — requires IOMMU DMA translation for its boot sequence. When identity mode bypasses translation, the FSP cannot complete its initialization, and the GPU fails to come online.

Approach 2: Runtime Unbind/Rebind

The second approach was more aggressive: try to unbind the nvidia driver from the GPUs at runtime, set the IOMMU groups to identity, and then rebind. This failed because Blackwell GPUs have a hardware-level lock: once the nvidia driver probes the GPU and initializes the FSP, the FSP locks its Write Protected Region (WPR2). Any subsequent unbind and rebind attempt fails with kfspSendBootCommands_HAL error (code 0x62:0xffff:2142).

The assistant tested every available software reset mechanism — Function Level Reset (FLR), Secondary Bus Reset (SBR), CXL bus reset, and even gpu-admin-tools --reset-with-sbr. None of them cleared the FSP lock. The only way to recover was a full power cycle (reboot) followed by a PCI remove, SBR, and rescan sequence — and even that only worked if nvidia was already loaded when the rescan happened.

Approach 3: DmaRemapPeerMmio Tuning

The third approach was to try to make the existing DmaRemapPeerMmio=1 parameter work better. This nvidia driver parameter attempts to remap peer MMIO (Memory-Mapped I/O) regions to enable P2P access through the IOMMU. The assistant confirmed it was already enabled and tested its behavior, finding directional corruption: transfers from lower-numbered GPUs to higher-numbered GPUs worked, but the reverse direction produced corrupted data.

The team considered trying an undocumented parameter — NVreg_RegistryDwords="PeerMappingOverride=1" — but judged it too risky for a production system where another tenant was actively using the other four GPUs.

The Conclusion: A Hard No

After exhausting these approaches, the assistant reached a definitive conclusion: P2P DMA cannot be restored on this system. The IOMMU must remain in DMA-FQ mode because:

  1. amd_iommu=on (not iommu=pt) is required for SEV-SNP
  2. Identity IOMMU domains break Blackwell GPU FSP boot
  3. The nvidia driver's DmaRemapPeerMmio=1 parameter has directional corruption
  4. NCCL_P2P_DISABLE=1 must remain to force SHM transport This conclusion is notable for its clarity and finality. In a field where many problems have workarounds or partial solutions, the assistant identified a genuine hardware-software boundary that cannot be crossed with current technology. The Blackwell FSP's dependency on IOMMU translation is a design constraint that no amount of kernel configuration or driver parameter tuning can overcome.

Why This Message Was Written

Message 6428 is not a typical assistant response. It contains no tool calls, no commands to execute, no code to review. It is a comprehensive system state documentation — a written record of everything learned during the session, the current state of every component, and the decisions that were made.

There are several reasons why this message was written in this form:

1. Knowledge Preservation. The investigation into P2P DMA restoration was complex and spanned multiple reboots, kernel configurations, and hardware resets. Without thorough documentation, the reasoning behind the conclusion — and the evidence supporting it — would be lost. The message serves as a permanent record that can be referenced later if someone asks "why is P2P disabled?" or "did we try identity IOMMU domains?"

2. Decision Rationale. The decision to abandon P2P restoration is not obvious. A less thorough investigation might have concluded "it doesn't work" without understanding why. The message documents the root cause — Blackwell FSP boot requires IOMMU translation — so that future engineers understand that this is a fundamental hardware limitation, not a configuration oversight.

3. System State Snapshot. The message captures the exact state of the system at a point in time: which software versions are installed, which patches have been applied, which services are running, which files exist where. This is invaluable for debugging future issues, reproducing the environment, or understanding what changes have been made.

4. Boundary Definition. The message explicitly defines what was tried and what didn't work. The "What P2P Approaches Were Exhausted" section lists six distinct approaches that were tested and failed. This prevents future wasted effort — no one needs to retry identity IOMMU domains or runtime unbind/rebind because the message clearly states why they don't work.

5. Handoff Documentation. The system has multiple tenants and agents. The message serves as a handoff document, ensuring that anyone who interacts with this system in the future understands its constraints and current state.

The Reasoning and Decision-Making Process

While message 6428 is a summary rather than a transcript of reasoning, the thinking process is visible in its structure and content. Several key reasoning patterns emerge:

Systematic Hypothesis Testing

The assistant approached the P2P problem like a scientific investigation. Each hypothesis was formulated, tested, and either confirmed or refuted with empirical evidence. The progression is clear:

  1. Hypothesis: Per-IOMMU-group identity domains can bypass translation for specific GPUs. Test: Create modprobe hook, set identity before nvidia loads. Result: FSP boot fails with error 0x177. Conclusion: Blackwell FSP requires IOMMU translation.
  2. Hypothesis: Runtime unbind/rebind can work around the boot-time constraint. Test: Unbind nvidia, set identity, rebind. Result: FSP lock persists across all software resets. Conclusion: Blackwell GSP/FSP locks permanently on first probe.
  3. Hypothesis: DmaRemapPeerMmio=1 can be tuned to work correctly. Test: Test P2P transfers in both directions. Result: Directional corruption. Conclusion: Partial workaround is insufficient.

Root Cause Analysis

The assistant didn't stop at "it doesn't work." It traced the failure to a specific mechanism: the Blackwell GPU's Firmware Security Processor (FSP) requires IOMMU DMA translation for its boot sequence. The error code 0x177 from kfspSendBootCommands_GH100 is a specific, identifiable failure mode. This level of root cause analysis transforms a vague "P2P is broken" into a precise "Blackwell FSP boot fails with identity IOMMU domains."

Risk Assessment

The decision to remove the modprobe hook and revert to DMA-FQ mode involved risk assessment. The assistant considered:

Assumptions Made and Challenged

Message 6428 reveals several assumptions that were made during the investigation, some of which were validated and some of which were challenged.

Assumption: IOMMU identity domains are transparent to devices

This was the critical assumption that drove the entire investigation. The team assumed that setting an IOMMU group to identity mode would be transparent to the device — the GPU would simply see direct PCIe access without translation, and everything would work as if the IOMMU weren't there.

This assumption was proven wrong. The Blackwell FSP actively depends on IOMMU translation for its boot sequence. The FSP firmware apparently uses DMA transactions during initialization that require the IOMMU to translate addresses, and when identity mode bypasses translation, the FSP cannot complete its boot sequence.

This is a surprising finding. IOMMU identity domains are supposed to be transparent — they're designed to pass through DMA transactions unchanged. The fact that a GPU's firmware-level boot process depends on IOMMU translation suggests a deeper integration between the GPU firmware and the platform's memory management than is commonly understood.

Assumption: Software resets can clear GPU firmware state

The assistant assumed that one of the available software reset mechanisms (FLR, SBR, CXL bus reset) would clear the FSP lock state, allowing the GPU to be re-initialized after an unbind/rebind cycle. This assumption was proven wrong for Blackwell GPUs. The FSP lock survives all software resets, requiring a full power cycle to clear.

This is a significant finding for anyone managing Blackwell GPUs in virtualized or hot-plug environments. If a Blackwell GPU needs to be re-initialized, a full system reboot is required — no amount of software resetting will work.

Assumption: The modprobe install hook approach would work

The assistant assumed that intercepting modprobe nvidia with an install hook and setting identity domains before the driver loaded would be a clean solution. This assumption was partially validated — the hook did run at the right time and set the identity domains correctly. The failure was not in the mechanism but in the fundamental incompatibility between identity domains and the FSP boot process.

Assumption: DmaRemapPeerMmio=1 would provide symmetric P2P

The assistant assumed that if DmaRemapPeerMmio=1 worked at all, it would work symmetrically — P2P transfers would work in both directions. This assumption was proven wrong. The directional corruption (lower-to-higher GPU indices work, reverse fails) suggests an addressing or mapping asymmetry in the remapping logic.

Input Knowledge Required to Understand This Message

Message 6428 is dense with technical detail. To fully understand it, a reader needs knowledge of:

GPU Architecture: Understanding what Blackwell GPUs are (NVIDIA's latest architecture, compute capability 12.0/SM120), what GDDR7 memory is, and how PCIe Gen5 x16 interfaces work.

IOMMU and Virtualization: Understanding IOMMU translation vs. pass-through (identity) mode, DMA remapping, IOMMU groups, and how these interact with GPU P2P DMA. Knowledge of SEV-SNP (AMD's Secure Encrypted Virtualization) and why it requires full IOMMU translation.

NVIDIA Driver Stack: Understanding the nvidia driver module hierarchy (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm), the FSP/GSP firmware processor, WPR (Write Protected Regions), and NCCL (NVIDIA Collective Communications Library).

LLM Inference Infrastructure: Understanding tensor parallelism (TP), multi-token prediction (MTP/EAGLE speculation), SGLang architecture, model serving, and throughput benchmarking.

Linux Kernel and System Administration: Understanding modprobe install hooks, systemd services, PCI device management (bind/unbind, remove/rescan), IOMMU group sysfs interface, dmesg/journalctl log analysis, and Proxmox LXC/VM management.

Model Architecture: Understanding Mixture-of-Experts (MoE) models, the Qwen3.5 architecture (hybrid GDN with linear attention + full attention layers), GQA (Grouped Query Attention), and BF16 precision.

Without this background, much of the message's significance would be lost. The assistant is writing for an audience that shares this technical context — likely other engineers working on the same infrastructure or similar deployments.

Output Knowledge Created by This Message

Message 6428 creates significant output knowledge that extends beyond its immediate context:

1. Blackwell FSP Boot Requires IOMMU Translation: This is the most important finding. It establishes that Blackwell GPUs cannot be used with IOMMU identity domains — the FSP firmware depends on IOMMU translation during boot. This has implications for any deployment that needs to mix IOMMU translation requirements (e.g., SEV-SNP) with Blackwell GPUs.

2. Blackwell FSP Lock Survives All Software Resets: The finding that FLR, SBR, and CXL bus reset cannot clear the FSP lock means that Blackwell GPU re-initialization requires a full power cycle. This is critical for hot-plug, virtualization, and dynamic GPU reconfiguration scenarios.

3. DmaRemapPeerMmio Directional Corruption: The documentation that P2P transfers work in one direction but not the other is a specific, actionable finding that can guide future debugging or driver configuration.

4. Proven Non-Viability of Specific Approaches: The message explicitly documents six approaches that were tried and failed. This prevents future wasted effort and serves as a reference for anyone considering similar approaches.

5. Complete System State Documentation: The message provides a full inventory of software versions, patches, configurations, and file locations. This is invaluable for reproducibility, debugging, and knowledge transfer.

6. Benchmark Baselines: The throughput benchmarks (with and without MTP) provide performance baselines that can be used to evaluate future optimizations or regressions.

7. SM120 Backend Compatibility Matrix: The table showing which backends work on SM120 (Blackwell) is a practical reference for anyone deploying models on this architecture.

The Thinking Process Visible in the Message

While message 6428 is a summary rather than a transcript of real-time reasoning, the thinking process is embedded in its structure. Several patterns are evident:

Prioritization by Impact

The message prioritizes information by its importance and actionability. The critical discovery — "IOMMU Identity Domains BREAK Blackwell GPU FSP Boot" — is the first substantive section after the goal and instructions. It's labeled "CRITICAL" and presented with a numbered list of findings. This tells the reader what matters most.

Exhaustive Enumeration

The "What P2P Approaches Were Exhausted" section lists six approaches in order of sophistication. This enumeration serves multiple purposes: it demonstrates thoroughness, prevents future wasted effort, and provides a reference for anyone wondering "what about trying X?"

Causal Chain Documentation

The message traces causal chains from symptom to root cause. For example:

Explicit Boundary Setting

The message is careful to distinguish between what was tried and what wasn't. The "Possible Future Optimizations (Not Yet Done)" section explicitly lists things that could be tried but haven't been, with notes on why they weren't pursued (e.g., "undocumented — risky"). This boundary setting prevents confusion about what has been ruled out vs. what simply hasn't been tested.

Contextual Awareness

The message shows awareness of the broader context: the other tenant using the SEV-SNP VM, the need to not reboot without permission, the constraint of no swap space, the use of zsh on the container. This contextual awareness ensures that the documentation is actionable — it doesn't just describe what is, but what constraints shaped the decisions.

The Broader Implications

The findings documented in message 6428 have implications beyond this single Proxmox host:

For Blackwell GPU Deployments: Anyone deploying Blackwell GPUs in environments with IOMMU translation enabled (virtualized environments, confidential computing, secure enclaves) needs to be aware that P2P DMA may not work, and that identity IOMMU domains will break GPU initialization.

For SEV-SNP and GPU Coexistence: The requirement for full IOMMU translation in SEV-SNP environments creates a fundamental conflict with Blackwell GPU P2P DMA. This may require architectural decisions about workload placement — perhaps dedicating certain hosts to SEV-SNP workloads and others to GPU inference, rather than trying to combine them.

For NVIDIA Driver Development: The directional corruption in DmaRemapPeerMmio=1 and the FSP boot dependency on IOMMU translation are issues that NVIDIA may need to address in future driver releases. The message notes "Waiting for nvidia driver fix" as a possible future optimization.

For Infrastructure Documentation: The message serves as a model for how to document complex infrastructure investigations. It combines system state, experimental results, decision rationale, and actionable conclusions in a single document.

Conclusion

Message 6428 is far more than a status update. It is a comprehensive technical document that records a significant discovery about Blackwell GPU behavior, documents the systematic investigation that led to that discovery, and establishes the current state and constraints of a complex multi-GPU inference deployment.

The message's greatest strength is its clarity about boundaries. It doesn't just say "P2P doesn't work" — it explains why it doesn't work at multiple levels of causality: the IOMMU configuration required by SEV-SNP, the Blackwell FSP's dependency on IOMMU translation, the FSP lock that survives all software resets, and the directional corruption in the DmaRemapPeerMmio workaround. Each of these is a boundary — a point where the system's behavior is constrained by fundamental hardware or software design decisions.

For the engineers who will maintain and evolve this system, message 6428 is a map of the territory. It shows which paths have been explored and which are dead ends. It marks the safe routes (DMA-FQ mode, NCCL_P2P_DISABLE, MTP speculation) and the dangerous ones (identity IOMMU domains, undocumented driver parameters). It provides the context needed to make informed decisions about future optimizations.

And for the broader community of AI infrastructure engineers, the message offers a cautionary tale: even with the most powerful consumer GPUs and the most sophisticated software stack, there are hardware-level constraints that cannot be worked around. The Blackwell FSP's dependency on IOMMU translation is one such constraint — a reminder that the boundary between software and hardware is not always negotiable.