The Reconnaissance Before the Cut: Mapping IOMMU Groups for GPU P2P Restoration

In the long arc of optimizing LLM inference on a complex multi-GPU system, few moments are as charged as the transition from research to action. Message [msg 6248] captures precisely that threshold. After a deep research phase investigating whether per-group IOMMU identity domains could restore GPU P2P DMA on a Proxmox host running 8 NVIDIA RTX PRO 6000 Blackwell GPUs under full IOMMU translation, the assistant takes its first concrete step toward implementation. But rather than charging ahead blindly, it does something methodical and revealing: it checks the current state of the IOMMU groups on the host, verifying the mapping between GPU PCI addresses, IOMMU groups, drivers, and domain types before touching anything.

This message is the reconnaissance before the cut — the careful measurement that precedes any surgical intervention on a running production system.

Context: The P2P DMA Problem

To understand why this message matters, we need to understand the problem it addresses. The system in question is a Proxmox host (kpro6) running 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split 4+4: four GPUs on NUMA 0 bound to the nvidia driver for an LXC container running SGLang inference, and four GPUs on NUMA 1 bound to vfio-pci for a SEV-SNP confidential VM. The host boots with amd_iommu=on (full translation mode, not passthrough) because SEV-SNP requires it. This breaks GPU-to-GPU P2P DMA — the IOMMU has no mappings for GPU BAR addresses, so when one GPU tries to DMA directly to another GPU's memory, the IOMMU logs IO_PAGE_FAULTs and the data is corrupted. The current workaround is NCCL_P2P_DISABLE=1, forcing NCCL to use slower SHM transport instead of P2P DMA.

The user had asked directly: "Is there anything that can be done to get p2p going with iommu?" ([msg 6238]). The assistant researched the question in a subagent task ([msg 6247]) and discovered that since Linux kernel v5.11, the sysfs interface at /sys/kernel/iommu_groups/<grp_id>/type allows per-group switching between translation (DMA-FQ) and passthrough (identity) modes. This means the 4 nvidia GPU groups could be set to identity mode (effectively passthrough for those devices only) while keeping the 4 vfio-pci groups in DMA-FQ translation mode for SEV-SNP. The research concluded this was "very promising."

The Message: Three Bash Commands, Three Layers of Understanding

Message [msg 6248] opens with the assistant saying, "Excellent — the research is very promising. Let me also check the current state of the IOMMU groups on the host to understand exactly which groups correspond to which GPUs." This single sentence reveals the assistant's reasoning: the research has been completed, the approach is viable in theory, but before any changes can be made, the exact mapping between physical devices and IOMMU groups must be known. The assistant then issues three parallel bash commands, each probing a different layer of the system.

Command 1: The IOMMU Group Map

The first command iterates over all 8 GPU PCI addresses — the 4 nvidia-bound GPUs (0000:01:00.0, 0000:11:00.0, 0000:61:00.0, 0000:71:00.0) and the 4 vfio-pci-bound GPUs (0000:81:00.0, 0000:91:00.0, 0000:e1:00.0, 0000:f1:00.0). For each, it resolves the IOMMU group number via the symlink at /sys/bus/pci/devices/$gpu/iommu_group, identifies the bound driver, and reads the group's domain type.

The output is a clean, complete inventory:

Command 2: Kernel Version and Boot Parameters

The second command checks uname -r and /proc/cmdline. The kernel is 6.14.11-5-bpo12-pve — a Proxmox backport kernel based on 6.14. This is significant because the per-group IOMMU domain switching feature was introduced in kernel v5.11, so version 6.14 comfortably supports it. The boot cmdline confirms amd_iommu=on (full translation), mem_encrypt=on (memory encryption for SEV), and kvm_amd.sev=1 (SEV-SNP enabled) — all of which must remain unchanged.

This command serves as a sanity check. Before attempting to write identity to any sysfs file, the assistant verifies that the kernel is recent enough to support the feature. If the kernel were older than 5.11, the entire approach would be dead on arrival.

Command 3: IOMMU Fault History

The third command checks the amd_iommu_v2 module parameters (none exist on this system) and tails the recent dmesg output for IOMMU faults and SEV-related messages. The output shows the familiar IO_PAGE_FAULTs — the very problem being addressed. The faulting addresses (0x24000000000, 0x34000001000, etc.) are GPU BAR addresses, confirming that the faults occur when GPUs attempt P2P DMA through the IOMMU without proper mappings.

This command provides a baseline: before the change, the system is logging these faults continuously. After switching to identity mode, these faults should disappear — and the dmesg output will serve as the verification.

The Thinking Process: Methodical, Risk-Aware, Production-Minded

What makes this message particularly interesting is the thinking process it reveals. The assistant is not simply executing a plan; it is building a mental model of the system's current state before making any modifications. Several key assumptions and decisions are embedded in this approach:

Assumption 1: The sysfs interface will work as documented. The assistant assumes that writing identity to /sys/kernel/iommu_groups/<grp_id>/type will actually change the IOMMU domain type for that group. This is a reasonable assumption given the kernel documentation, but it's not yet verified — the type file might be read-only, or the kernel might reject the write for security reasons.

Assumption 2: Changing the domain type for nvidia groups won't affect the vfio-pci groups. The assistant assumes that IOMMU groups are truly independent — that switching group 42 to identity mode won't somehow cascade to group 101. This is supported by the architecture (each group has its own domain), but it's an assumption that will need to be validated.

Assumption 3: The SEV-SNP VM won't be affected. The assistant assumes that SEV-SNP only checks for amd_iommu=on at the global level, not per-group. This is a critical assumption because the SEV-SNP VM (VM 132, snp-sandbox) is currently running and must not be disrupted.

Assumption 4: The nvidia driver can be unbound and rebound without leaving the GPUs in a broken state. This is perhaps the most operationally risky assumption. Unbinding a GPU from the nvidia driver while it's initialized requires careful sequencing, and rebinding might fail if the driver doesn't properly re-initialize the device.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains:

  1. Linux IOMMU architecture: Understanding what IOMMU groups are, how DMA-FQ vs identity modes differ, and how the sysfs interface exposes per-group domain types. The DMA-FQ type indicates DMA with a flush queue — the standard translation mode where the IOMMU translates all DMA addresses. identity mode bypasses translation, effectively making the device direct-mapped.
  2. PCIe topology and device addressing: The PCI bus:device:function notation (0000:01:00.0 means bus 0, device 1, function 0 on domain 0) and how IOMMU groups map to physical PCIe hierarchy.
  3. NVIDIA GPU driver model: How the nvidia driver binds to PCI devices, how nvidia-persistenced manages GPU state, and the relationship between driver binding and GPU initialization.
  4. SEV-SNP and AMD IOMMU requirements: Understanding that SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) requires amd_iommu=on in full translation mode, and why iommu=pt (passthrough) is incompatible.
  5. NCCL and GPU P2P DMA: How NCCL uses GPU direct P2P transfers for all-reduce operations, and why IOMMU translation breaks this path.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. Complete IOMMU group-to-GPU mapping: Groups 42, 61, 28, 10 → nvidia GPUs on NUMA 0; groups 101, 117, 90, 72 → vfio-pci GPUs on NUMA 1. All currently in DMA-FQ mode.
  2. Kernel version confirmation: 6.14.11 supports per-group identity domains (feature since 5.11).
  3. Boot configuration snapshot: The kernel cmdline includes amd_iommu=on, mem_encrypt=on, kvm_amd.sev=1 — all required for SEV-SNP and none of which will be changed.
  4. Baseline IOMMU fault pattern: The dmesg output shows ongoing IO_PAGE_FAULTs with GPU BAR addresses, confirming the P2P DMA problem is active.
  5. Confirmation of clean group isolation: Each GPU is in its own IOMMU group with no other PCI functions (audio devices, USB controllers, etc.) sharing the group. This means switching a group to identity mode affects only that single GPU.

Why This Message Matters

In the broader narrative of the coding session, message [msg 6248] is the pivot point. The previous ~40 segments of work had established the environment, resolved countless build issues, deployed models, and diagnosed the P2P problem. The research task in [msg 6247] had identified a potential solution. But between theory and practice lies the messy reality of a running production system with multiple tenants, live VMs, and expensive hardware that cannot be rebooted casually.

This message is where the assistant bridges that gap. It doesn't assume the IOMMU group numbers — it reads them from the live system. It doesn't assume the kernel supports the feature — it checks the version. It doesn't assume the fault pattern is what was previously reported — it reads the latest dmesg output. Every assumption is validated against the current state of the system before any change is made.

The three bash commands in this message are, in essence, a pre-flight checklist. They answer three questions: Where are we? What kernel are we running? What does the current fault landscape look like? With these answers, the assistant can proceed to the next step — actually unbinding the GPUs, setting identity domains, and rebinding — with confidence that the target is correctly identified and the approach is viable.

For anyone working with complex multi-GPU systems, IOMMU configuration, or production infrastructure, this message exemplifies the right approach to risky system changes: measure first, cut second. The reconnaissance is not optional — it is the foundation on which every subsequent decision rests.