The Moment the P2P Dream Died: Verifying IOMMU Identity Domains After Reboot

Introduction

In the long and winding journey of deploying NVIDIA Blackwell GPUs for production inference, few moments are as pivotal — and as quietly ominous — as message 6370 in this opencode session. The message itself is deceptively brief: a simple statement of intent to verify a boot-time configuration, accompanied by a todo list update. But in the narrative arc of the conversation, it marks the exact inflection point where a carefully laid plan to restore GPU peer-to-peer (P2P) DMA begins to unravel, ultimately leading to the discovery that Blackwell's Firmware Security Processor (FSP) is fundamentally incompatible with IOMMU identity domains.

The Message

The assistant writes:

Let me check if the IOMMU identity domains were set at boot and verify P2P status.

Then follows a todo list with items for verifying the identity domains, testing P2P DMA transfers, and conditionally removing the NCCL_P2P_DISABLE=1 workaround if P2P works.

On its surface, this is a routine post-reboot verification step. But to understand why this message matters, we must understand the immense weight of context that precedes it.

The Context: A Desperate Need for P2P DMA

The system in question is a Proxmox host with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs for inference) and a SEV-SNP VM (4 GPUs for other workloads). The inference workload — serving a Qwen3.5-122B-A10B model with SGLang using tensor parallelism (TP=4) — critically depends on NCCL's ability to perform P2P DMA transfers between GPUs for efficient all-reduce communication.

However, a fundamental conflict exists: SEV-SNP (Secure Encrypted Virtualization) requires the AMD IOMMU to be in full translation mode (amd_iommu=on), which normally prevents GPU P2P DMA because the IOMMU remaps DMA addresses. The previous session ([msg 6340]–[msg 6368]) had painstakingly worked around this by exploiting a Linux kernel feature: per-IOMMU-group domain type switching. The plan was to set the IOMMU groups containing the NUMA0 GPUs (the ones used by the LXC container) to identity mode — bypassing translation for those specific devices — while leaving the NUMA1 GPU groups (used by the VM) in DMA-FQ translation mode. This would theoretically restore P2P DMA for the inference workload without breaking SEV-SNP for the VM.

The assistant had created a boot-time service (gpu-iommu-identity.service) that would run before the NVIDIA driver loaded, unbind the NUMA0 GPUs, set their IOMMU groups to identity, and then let the NVIDIA driver bind to them fresh. The user had rebooted the host to test this configuration. Message 6370 is the assistant's response to the user's command: "rebooted, check P2P status" ([msg 6369]).

The Assumptions Embedded in This Message

Message 6370 is built on several critical assumptions, all of which will prove incorrect:

Assumption 1: The boot-time service would execute in the correct order. The gpu-iommu-identity.service was configured with Before=nvidia-persistenced.service, assuming this would guarantee it ran before the NVIDIA driver touched the GPUs. The assistant had not anticipated that the gpu-vfio-split.service — which registers the GPU PCI ID with vfio-pci — might run afterward and steal the NUMA0 GPUs.

Assumption 2: Setting identity domains at boot would be sufficient. The entire strategy depended on the idea that if the IOMMU group was in identity mode before the NVIDIA driver first probed the GPU, the driver would initialize correctly and P2P would work. This assumed that the NVIDIA driver's Blackwell initialization path is compatible with identity IOMMU domains.

Assumption 3: The Blackwell FSP state could be reset. The previous session had already discovered that the Blackwell GPU's Firmware Security Processor (FSP) survives driver unbind/rebind cycles, causing the kfspSendBootCommands_HAL failure with error 0x62:0xffff:2142. The boot-time approach was designed to avoid this entirely by setting identity before the first probe. The assumption was that a fresh probe with identity domains would succeed where a rebind had failed.

Assumption 4: The verification would be straightforward. The todo list shows a linear, optimistic plan: verify identity domains → test P2P → remove workarounds → restart SGLang. There is no contingency for the scenario where identity domains are set but the GPUs are bound to the wrong driver, or where identity domains themselves cause GPU initialization to fail.

The Reasoning: Why This Message Exists

The message exists because the assistant is following a systematic engineering methodology: make a change, reboot, verify. The user's instruction "rebooted, check P2P status" is a prompt to execute the verification phase of a test cycle. The assistant's response is the first step in that verification — it must check whether the boot-time configuration was applied correctly before proceeding to the actual P2P test.

The todo list reveals the assistant's mental model of the verification process:

  1. Verify IOMMU identity domains — check that the sysfs entries show type=identity for the NUMA0 GPU groups
  2. Test P2P DMA transfers — presumably using a CUDA P2P bandwidth test or similar
  3. Conditionally remove workarounds — if P2P works, remove NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce from the SGLang service configuration
  4. Restart SGLang with P2P enabled — the ultimate goal This is a rational, step-by-step plan. But it assumes the first step will succeed, which it will not — at least, not in the way expected.

What Actually Happens Next

The subsequent messages ([msg 6371]–[msg 6399]) reveal a cascade of failures:

  1. Identity domains are set correctly — but the NUMA0 GPUs are bound to vfio-pci instead of nvidia ([msg 6371]). The gpu-vfio-split.service ran after the identity service and grabbed all GPUs with matching PCI ID.
  2. Fixing the boot order — the assistant updates the identity script to also rebind GPUs to nvidia after vfio-split runs ([msg 6373]–[msg 6374]).
  3. The GSP dies on rebind — manually running the script unbinds the GPUs from vfio-pci and rebinds to nvidia, but nvidia-smi reports "No devices were found" ([msg 6376]). The GSP firmware state is corrupted by the unbind/rebind cycle.
  4. SBR doesn't help — even Secondary Bus Reset (SBR) on the parent PCI bridges fails to clear the GSP state ([msg 6382]). The error _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset) with WPR2 already up confirms the FSP state survives hardware reset.
  5. A brief success — the assistant discovers that if NVIDIA module is loaded before PCI rescan, the fresh probe succeeds ([msg 6383]). But the IOMMU groups default to DMA-FQ because nvidia's modalias-triggered loading happens before identity can be set.
  6. The modprobe hook approach — the assistant creates a modprobe install hook that sets identity domains before the real nvidia module loads ([msg 6393]–[msg 6398]). But this also fails — the GPUs still show "No devices were found" ([msg 6399]).
  7. The fatal discovery — eventually, the assistant discovers that even with the modprobe hook working correctly (identity set before nvidia loads), the Blackwell FSP boot sequence fails with error code 0x177 when IOMMU is in identity mode. The FSP requires specific DMA mappings set up by the kernel's DMA API in translation mode, and identity mode breaks this initialization. This is a hardware-level incompatibility — no amount of software reordering can fix it.

The Knowledge Required to Understand This Message

To fully grasp message 6370, one needs knowledge spanning several domains:

IOMMU internals: Understanding that the AMD IOMMU can operate in different domain types — DMA-FQ (full translation with flush queue) and identity (pass-through, no translation). The per-group switching feature is a relatively recent Linux kernel capability (6.14+).

NVIDIA GPU architecture: Understanding the Blackwell GPU's GSP (GPU System Processor) and FSP (Firmware Security Processor) — the embedded firmware that manages GPU initialization, power management, and security. The FSP runs independently of the host driver and maintains state across driver reloads.

PCIe topology and reset mechanisms: Understanding the difference between Function-Level Reset (FLR), Secondary Bus Reset (SBR), and CXL bus reset, and why SBR doesn't clear FSP state on Blackwell.

Virtualization and IOMMU: Understanding SEV-SNP (Secure Nested Paging) and why it requires IOMMU translation mode, creating the fundamental conflict with GPU P2P DMA.

Linux device driver model: Understanding how PCI drivers bind to devices, how modprobe install hooks work, and how the kernel's modalias mechanism triggers automatic module loading during PCI rescan.

The Output Knowledge Created

Message 6370 itself doesn't produce new knowledge — it's a statement of intent. But it initiates the verification process that produces the following critical knowledge:

  1. Boot-time service ordering matters — the gpu-iommu-identity.service must run after gpu-vfio-split.service, not before, because vfio-pci's new_id registration grabs all unbound GPUs.
  2. Blackwell FSP is incompatible with IOMMU identity mode — this is the most significant finding. The FSP requires DMA translation mappings that only exist when the IOMMU is in DMA-FQ mode. Identity mode deprives the FSP of these mappings, causing initialization to fail with error 0x177.
  3. No software reset can clear the FSP state — FLR, SBR, CXL bus reset, driver unbind/rebind — none of these reset the Blackwell FSP. The only way to clear it is a full platform power cycle (AC power off/on).
  4. The DmaRemapPeerMmio=1 parameter is the only remaining option — but it produces incomplete IOMMU mappings, with some peer pairs working and others faulting.

Mistakes and Incorrect Assumptions

The most significant mistake in message 6370 is not in what it says, but in what it doesn't say: there is no acknowledgment of the possibility that identity domains might be fundamentally incompatible with Blackwell GPUs. The todo list treats the verification as a routine check, not as a potentially fatal test of a critical assumption.

The assistant assumed that the boot-time approach would work because it avoided the driver rebind problem that plagued the runtime approach in the previous session. But this assumption was never validated against the actual hardware behavior. The Blackwell FSP's requirement for DMA translation mode was unknown at the time, and there was no way to discover it except through the experimental process that message 6370 initiates.

A more subtle mistake is the optimistic service ordering. The assistant had created the identity service to run before nvidia-persistenced, but hadn't considered that gpu-vfio-split.service (which was already in the boot sequence from earlier configuration) would interfere. This is a classic systems integration issue — multiple independent services interact in unexpected ways.

The Thinking Process Visible in the Message

While message 6370 is short, the todo list reveals the assistant's structured thinking. The items are ordered by dependency: verification must precede action, and conditional actions depend on verification results. The use of "If P2P works" as a condition shows that the assistant is thinking in terms of branching paths, even though the message doesn't explicitly discuss the failure case.

The brevity of the message is itself telling. The assistant is eager to get to the verification — it has been planning this reboot test for an entire session, and now the moment has arrived. The message reads like a developer taking a deep breath before running a critical test: "Let me check..." There's an undercurrent of anticipation, even hope, that after all the work of the previous session, P2P DMA will finally work.

Conclusion

Message 6370 is a quiet turning point in a much larger story. It is the moment before the discovery, the pause before the bad news. In the narrative of this coding session, it represents the last moment when the P2P DMA restoration plan still seems viable. Within minutes, the assistant will discover that the NUMA0 GPUs are bound to vfio-pci instead of nvidia, and within hours, it will discover that Blackwell's FSP simply cannot work with IOMMU identity domains — a hardware limitation that no amount of software engineering can overcome.

The message serves as a reminder that in systems engineering, the most carefully laid plans can fail at the point of contact with reality. The verification step is not merely a formality — it is where assumptions meet hardware, and where the true behavior of the system is revealed. Message 6370, for all its brevity, captures that moment perfectly: the engineer, ready to verify, unaware that the answer will be "no."