The IOMMU Identity Domain Investigation: A Systematic Research Campaign for GPU P2P DMA on Mixed-Trust Systems

Introduction

In the complex landscape of modern GPU-accelerated computing, few challenges are as technically demanding as enabling peer-to-peer (P2P) DMA between GPUs while maintaining the full security guarantees of an IOMMU in translation mode. This article examines a concentrated research campaign conducted by an AI assistant over the course of a multi-round opencode session, investigating whether per-IOMMU-group identity domains could restore GPU P2P DMA on a system with 8 NVIDIA RTX PRO 6000 Blackwell GPUs while preserving the security requirements of an AMD SEV-SNP confidential virtual machine.

The system under investigation was a Proxmox host with a complex split configuration: 4 GPUs on NUMA node 0 bound to the nvidia proprietary driver for LXC container inference workloads, and 4 GPUs on NUMA node 1 bound to the vfio-pci driver for an SEV-SNP confidential VM. The SEV-SNP VM imposed a hard constraint: the AMD IOMMU must remain in full translation mode (amd_iommu=on), ruling out the simpler iommu=pt (global passthrough) approach that would ordinarily resolve P2P DMA issues. The core problem was that GPU-to-GPU DMA was triggering IO_PAGE_FAULT errors because the IOMMU had no mappings for GPU BAR (Base Address Register) regions when one GPU attempted to DMA directly to another's BAR space.

The assistant's research spanned multiple rounds of web searches, documentation fetches, kernel source code analysis, and ultimately a real-world test that revealed a fundamental hardware limitation. This article synthesizes that work, tracing the arc from initial exploration through targeted verification to the final, unexpected discovery.

The Problem: A Tension Between Security and Performance

The system's architecture created an inherent tension. AMD's SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) is a confidential computing technology that encrypts VM memory and provides integrity protection against host-initiated DMA attacks. The SEV-SNP firmware checks that the IOMMU is actively providing DMA translation — it cannot operate with iommu=pt (global passthrough) because that would bypass the IOMMU's memory protection entirely. As documented in the SUSE SEV-SNP host setup guide, the IOMMU must be configured in non-passthrough mode for SEV-SNP to function [4][7].

However, with the IOMMU in full translation mode, GPU P2P DMA was broken. When GPU A attempted to DMA directly to GPU B's BAR memory region, the IOMMU intercepted the transaction and raised an IO_PAGE_FAULT because it had no page table entry mapping that physical address for GPU A's IOVA space. The nvidia proprietary driver, for reasons rooted in its internal architecture, did not install the necessary IOMMU mappings for inter-GPU BAR access in translation mode. This left the user in a bind: either disable P2P (losing performance) or disable IOMMU translation (breaking SEV-SNP security).

The proposed solution was elegant in concept: use the Linux kernel's per-IOMMU-group domain type switching feature to set the 4 nvidia-bound GPU groups to identity (passthrough) mode while keeping the 4 vfio-bound GPU groups in DMA (translation) mode. This would, in theory, allow P2P DMA between the identity-mode GPUs while preserving IOMMU protection for the SEV-SNP VM's devices.

The Research Campaign: A Methodical Multi-Round Investigation

The assistant's investigation unfolded across multiple messages, each building on the results of the previous round. The research methodology was systematic: start broad, identify gaps, drill into specifics, validate critical constraints, and finally synthesize.

Round 1: Broad Exploration

The opening move ([msg 1]) launched three parallel web searches covering the three dimensions of the problem: the kernel's IOMMU group type sysfs interface, AMD IOMMU per-group identity domain support, and NVIDIA GPU P2P behavior with IOMMU [3]. These searches cast a wide net, returning the kernel documentation page for the /sys/kernel/iommu_groups/ ABI, an unanswered Stack Overflow question about AMD IOMMU IO_PAGE_FAULTs, and NVIDIA's GPUDirect RDMA documentation. The assistant established the landscape: the per-group type switching feature existed, but its practical details and compatibility with the specific hardware stack were unknown.

Round 2: Drilling into Kernel Mechanics

The second round ([msg 2]) narrowed the focus with three more targeted queries [11]. The assistant searched for the kernel source code implementing iommu_group_store_type in drivers/iommu/iommu.c, probed the interaction between iommu.passthrough=1 and vfio-pci with SEV-SNP, and looked for NVIDIA driver-specific behavior with P2P in identity mode. This round demonstrated the assistant's understanding of where to find authoritative information: kernel source code on GitHub for implementation details, conference presentations for design intent, and developer forums for real-world reports.

Round 3: Primary Source Verification

The third round ([msg 3]) represented a critical pivot from broad searches to targeted, authoritative source retrieval [5]. The assistant fetched three specific URLs: the LWN.net article on "iommu: Per-group default domain type" (the kernel community's announcement of the feature), the kernel's official ABI documentation for the sysfs interface, and the kernel parameters reference page. This was a deliberate strategy to verify against primary sources rather than relying on second-hand summaries or forum posts.

Rounds 4-6: Filling Knowledge Gaps

Subsequent rounds continued the pattern of targeted investigation. The assistant searched for the specific unbind/rebind requirement for changing IOMMU group types ([msg 4]) [6], investigated NVIDIA's DGX OS documentation and SEV-SNP security requirements in parallel ([msg 5]) [10], and fetched the XDC 2020 presentation on P2P DMA challenges ([msg 6]) [2]. Each round addressed a specific knowledge gap identified from the previous results.

Round 7: The Critical Pivot

A pivotal moment came in [msg 7], where the assistant explicitly narrowed its focus to two critical questions: the behavior of iommu.passthrough=1 with VFIO, and whether AMD SEV-SNP absolutely requires IOMMU DMA translation [9]. The assistant recognized that these were the make-or-break constraints — if iommu.passthrough=1 could achieve the same effect as per-group identity while preserving VFIO translation, it would be a simpler solution. And if SEV-SNP could tolerate identity mode under any circumstances, the problem might be easier than anticipated.

Round 8: Targeted Source Verification

The assistant then fetched two specific sources ([msg 8]): the AMDSEV GitHub issue #88 about IOMMU passthrough compatibility with SEV-SNP, and a Proxmox forum thread about selective IOMMU passthrough [1]. These were the most authoritative sources available for the specific configuration scenario.

Round 9: The Moment of Synthesis

In [msg 9], the assistant made a rare and revealing meta-cognitive declaration: "I now have enough information to compile a comprehensive research summary. Let me also verify one more critical detail about SEV-SNP and IOMMU requirements" [8]. This message is a window into the assistant's self-assessment — it knew what it knew, and it knew what it still needed to verify. The final verification targeted the Confidential Containers project's SEV-SNP host setup guide, which provided the definitive confirmation that SEV-SNP requires non-passthrough IOMMU mode.

Round 10: The Comprehensive Synthesis

The culmination of this research was [msg 10], a comprehensive research summary that directly answered all six of the user's questions [7]. The message was structured as a formal technical report with six numbered sections, a concrete implementation script, and a risk assessment.

The key findings were:

  1. Per-group IOMMU identity domains are supported since kernel v5.11 via the /sys/kernel/iommu_groups/*/type sysfs interface, accepting values DMA, DMA-FQ, identity, and auto.
  2. The critical operational constraint: all devices in a group must be unbound from their driver before the domain type can be changed, requiring a careful unbind/change/rebind sequence.
  3. AMD IOMMU compatibility: the AMD IOMMU driver supports per-group identity domains with no known blockers, provided amd_iommu=force_isolation is not set.
  4. NVIDIA driver behavior: in identity mode, IOVA equals physical address, so GPU-to-GPU DMA targeting BAR addresses passes through the IOMMU without translation, eliminating IO_PAGE_FAULTs. However, nvidia-drm with modeset=1 can complicate the unbinding process.
  5. SEV-SNP compatibility: the SEV-SNP firmware checks the global IOMMU initialization mode, not per-group overrides. As long as amd_iommu=on is set globally, per-group identity domains do not trigger SEV-SNP safety checks.
  6. Alternative approaches rejected: iommu=pt is incompatible with SEV-SNP; nvidia module parameters (NVreg_EnablePCIeRelaxedOrderingMode, etc.) do not solve the core IOMMU mapping issue. The assistant provided a concrete bash script (/usr/local/bin/gpu-iommu-identity.sh) that automated the unbind/change/rebind process, with the recommendation to run it before the nvidia driver loads using a softdep or module load ordering mechanism.

The Unexpected Outcome: Blackwell FSP Boot Failure

The research phase concluded with a clear, well-supported recommendation. But when the approach was actually tested — after implementing the modprobe install hook and rebooting — a critical discovery emerged that no amount of web research could have predicted.

The NVIDIA Blackwell GPU's Firmware Security Processor (FSP) boot sequence failed with error code 0x177 when the IOMMU was in identity mode. The FSP apparently requires specific DMA mappings set up by the kernel's DMA API in translation mode during its initialization sequence. Identity mode breaks this initialization, and no software-level reset — not Function Level Reset (FLR), not Secondary Bus Reset (SBR), not even a CXL bus reset — can clear the resulting error state.

This finding was definitive: per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs. The approach cannot work regardless of timing, because the FSP boot sequence occurs early in the device initialization process, before any software intervention can establish the necessary DMA mappings.

The assistant immediately reverted the change by removing the modprobe hook and rebooting, restoring the working DMA-FQ IOMMU configuration. The MTP (Multi-Token Prediction) speculation that had been enabled in a previous session survived the reboot and continued to provide significant throughput improvements of 12-45% per request at low concurrency.

The Broader Significance

This investigation, though it ultimately produced a negative result, demonstrates several important principles for systems engineering research:

Systematic methodology matters. The assistant's layered approach — broad exploration, targeted verification, primary source validation, constraint checking, and finally synthesis — is a model for tackling complex, multi-dimensional technical problems. Each round built on the previous one, and the assistant consistently identified the critical unknowns before proceeding.

Primary sources are essential. The assistant's insistence on fetching kernel documentation, LWN articles, and official vendor guides rather than relying on search snippets or forum posts ensured that the research conclusions were grounded in authoritative information. This was particularly important for the SEV-SNP requirement, where the SUSE documentation provided the definitive answer.

Hardware limitations can override software solutions. The Blackwell FSP boot failure is a hardware-level constraint that no amount of software configuration could circumvent. This underscores the importance of empirical testing even when research suggests a solution should work. The assistant's research was thorough and correct in its analysis of the kernel features, but it could not anticipate a hardware firmware limitation that was not documented anywhere.

Negative results are valuable knowledge. Discovering that per-group IOMMU identity domains cannot work with Blackwell GPUs saves countless hours of future troubleshooting. It forces the search for alternative approaches — in this case, the nvidia driver's DmaRemapPeerMmio=1 parameter, which was already enabled but producing incomplete IOMMU mappings, with some peer pairs working and others faulting.

Conclusion

The IOMMU identity domain investigation spanned multiple rounds of systematic research, from broad exploration to targeted verification to comprehensive synthesis. The assistant demonstrated sophisticated research methodology: identifying knowledge gaps, prioritizing authoritative sources, validating critical constraints, and synthesizing findings into actionable recommendations. The ultimate discovery — that Blackwell's FSP boot sequence fails under identity mode — was a hardware limitation that no amount of software research could have predicted, but the research process itself was sound and valuable.

The system remains stable with SGLang serving Qwen3.5-122B-A10B BF16 on 4 GPUs with TP=4, MTP speculation delivering 12-45% throughput improvement, and NCCL_P2P_DISABLE=1 working around the P2P limitation. The remaining avenue for P2P restoration is the nvidia driver's DmaRemapPeerMmio=1 parameter, which requires further investigation to resolve the incomplete IOMMU mappings that currently cause some peer pairs to fault while others work correctly.## References

[1] "Drilling into the Details: How a Targeted Web Fetch Resolved the IOMMU Passthrough Dilemma" — Analysis of message 8's strategic web fetches to AMDSEV GitHub issue #88 and Proxmox forum thread.

[2] "The Critical Fetch: How One Research Message Uncovered the Key to GPU P2P DMA with IOMMU" — Examination of message 6's PDF fetch and Exa search for the XDC 2020 P2P DMA presentation.

[3] "The Opening Move: How an AI Assistant Begins Investigating GPU P2P DMA with IOMMU Per-Group Identity Domains" — Analysis of message 1's three parallel search queries launching the research campaign.

[4] "The Per-Group IOMMU Identity Domain Question: A Deep Dive into GPU P2P DMA on Mixed-Trust AMD Systems" — The user's initial research request (message 0) posing six detailed research questions.

[5] "The Research Turn: Fetching Kernel Documentation to Solve GPU P2P DMA with IOMMU" — Analysis of message 3's pivot to primary source verification via LWN article and kernel documentation.

[6] "Drilling Down: The Search for Per-Group IOMMU Identity Domains" — Examination of message 4's targeted searches for iommu_group_store_type implementation details.

[7] "Navigating the IOMMU Tightrope: Per-Group Identity Domains for GPU P2P DMA Under Full Translation" — Analysis of message 10's comprehensive research synthesis with implementation script and risk assessment.

[8] "The Moment of Synthesis: How an AI Assistant Decides It Has Enough Information" — Analysis of message 9's meta-cognitive declaration and final SEV-SNP verification.

[9] "The Pivot Point: Investigating iommu.passthrough and VFIO Compatibility in GPU P2P DMA Research" — Analysis of message 7's targeted investigation of iommu.passthrough=1 and SEV-SNP requirements.

[10] "The Parallel Search: Unraveling GPU P2P DMA and IOMMU Identity Domains" — Analysis of message 5's parallel investigation of NVIDIA driver behavior and SEV-SNP security risks.

[11] "Drilling Down: The Assistant's Targeted IOMMU Research Queries" — Analysis of message 2's second-wave targeted searches into kernel source code and NVIDIA driver specifics.