The Diagnostic Dead End: Probing IOMMU Group Types and NVIDIA Driver Parameters for GPU P2P DMA

In the middle of a high-stakes machine learning deployment spanning Proxmox virtualization, SEV-SNP confidential computing, and eight RTX PRO 6000 Blackwell GPUs, a single bash command — message [msg 6243] — captures the moment when a promising debugging thread runs into a wall. The message is deceptively simple: the assistant runs a one-liner over SSH to query IOMMU group types and NVIDIA driver parameters on the Proxmox host. But this command represents the culmination of a multi-step diagnostic journey into why GPU-to-GPU peer-to-peer (P2P) DMA breaks under AMD IOMMU full translation mode, and the beginning of the realization that no software-level knob can fix it.

The Context: A Broken P2P DMA Pipeline

The story begins with a Proxmox host running an SEV-SNP (Secure Encrypted Virtualization) configuration. The host's kernel command line includes amd_iommu=on and mem_encrypt=on kvm_amd.sev=1, which together enable AMD's IOMMU in full translation mode — not passthrough mode. This distinction is critical: in passthrough mode (iommu=pt), the IOMMU allows devices to DMA directly to each other's memory-mapped I/O (MMIO) regions without translation. In full translation mode, every DMA transaction goes through the IOMMU page tables, which only contain mappings for system RAM — not for other devices' PCIe BAR (Base Address Register) regions.

When the assistant first deployed the Qwen3.5-122B BF16 model across 4 GPUs using NCCL (NVIDIA Collective Communications Library), the server hung during init_torch_distributed. The root cause, diagnosed over several messages ([msg 6216] through [msg 6222]), was that every GPU-to-GPU P2P transfer produced corrupted data. A CUDA copy test showed "MISMATCH" for every pair of GPUs, and dmesg was filled with IO_PAGE_FAULT events from the AMD-Vi IOMMU. The fix was pragmatic: set NCCL_P2P_DISABLE=1, forcing NCCL to fall back to shared memory (SHM) transport, which bypasses GPU BAR-to-BAR DMA entirely.

But the user was not satisfied with this workaround. In [msg 6238], they asked: "Is there anything that can be done to get p2p going with iommu? Any bios settings about iommu coverage, related settings?" This question sets the stage for the subject message.

The Message: A Focused Probe of Kernel and Driver State

Message [msg 6243] is a single bash command executed on the Proxmox host (root@10.1.2.6):

ssh root@10.1.2.6 'for dev in 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0; do echo -n "$dev: "; cat /sys/bus/pci/devices/$dev/iommu_group/type 2>/dev/null; done; echo "---"; cat /sys/module/nvidia/parameters/NVreg_EnablePCIeRelaxedOrderingMode 2>/dev/null; echo "---"; cat /sys/module/nvidia/parameters/NVreg_RegistryDwords 2>/dev/null'

The command probes three things:

  1. IOMMU group type for each of the four NVIDIA GPUs (PCIe addresses 0000:01:00.0, 0000:11:00.0, 0000:61:00.0, 0000:71:00.0). The iommu_group/type sysfs file reports whether the IOMMU group uses "DMA" (strict DMA translation) or "DMA-FQ" (DMA with fine-grained translation and queuing). This is relevant because the IOMMU translation mode determines whether devices in the same group can DMA to each other.
  2. NVreg_EnablePCIeRelaxedOrderingMode — a NVIDIA kernel module parameter that controls whether the driver enables PCIe relaxed ordering, which can affect P2P DMA performance and correctness in certain topologies.
  3. NVreg_RegistryDwords — a catch-all NVIDIA driver parameter for injecting arbitrary registry-style configuration values, sometimes used to enable experimental or platform-specific GPU features. The output is:
0000:01:00.0: DMA-FQ
0000:11:00.0: DMA-FQ
0000:61:00.0: DMA-FQ
0000:71:00.0: DMA-FQ
---
---

All four GPUs report "DMA-FQ" as their IOMMU group type. Both NVIDIA driver parameters are empty — neither relaxed ordering mode nor any registry dwords have been configured.## Interpreting the IOMMU Group Type: DMA-FQ

The "DMA-FQ" designation is significant. In the Linux IOMMU subsystem, IOMMU groups can be typed as "DMA" (direct memory access with full translation) or "DMA-FQ" (DMA with fine-grained translation and queuing). The "FQ" suffix indicates that the IOMMU supports fine-grained address translation with a command queue — a feature of modern AMD IOMMU implementations. However, neither "DMA" nor "DMA-FQ" implies passthrough mode. Both enforce full IOMMU translation for all device DMA transactions.

The critical observation here is that all four GPUs are in different IOMMU groups (groups 42, 61, 28, and 10 respectively, as revealed in [msg 6242]). When devices are in separate IOMMU groups, the IOMMU enforces isolation between them — each group has its own page tables, and DMA from one group to another must go through the IOMMU translation layer. For GPU P2P DMA to work, the IOMMU would need to have mappings for each GPU's BAR regions in the other GPUs' page tables, which it does not by default.

The fact that all four report "DMA-FQ" confirms that the IOMMU is operating in full translation mode for every GPU. There is no per-device passthrough override visible at this level. The iommu_group/type sysfs file does not expose a "passthrough" variant — passthrough is typically configured at the IOMMU domain level via kernel boot parameters (iommu=pt or amd_iommu=on iommu=pt) or via per-device ACS (Access Control Services) overrides.

The Empty NVIDIA Parameters: No Hidden Knobs

The second and third probes return empty results. NVreg_EnablePCIeRelaxedOrderingMode is unset (defaulting to the driver's internal default, typically 0 or disabled), and NVreg_RegistryDwords is empty. This tells the assistant that no NVIDIA driver-level workarounds have been applied — and more importantly, that no known driver parameter exists to bypass IOMMU translation for P2P DMA.

This is a dead end. If there were a driver parameter to enable P2P DMA through the IOMMU (such as enabling ACS override or requesting passthrough mappings), it would likely be configured through NVreg_RegistryDwords or a dedicated module parameter. The empty output confirms that no such escape hatch is available in the current driver version.

The Reasoning Behind the Probe

The assistant's thinking process, visible in the preceding messages, follows a logical chain:

  1. Problem identification ([msg 6216]): Every GPU-to-GPU P2P transfer produces data mismatches. The IOMMU is corrupting DMA transactions.
  2. Root cause analysis ([msg 6217]): The IOMMU in full translation mode has no mappings for GPU BAR regions. When GPU A tries to DMA to GPU B's BAR, the IOMMU generates an IO_PAGE_FAULT and the transaction delivers corrupted data or zeros.
  3. Workaround ([msg 6221]): Disable NCCL P2P entirely, forcing shared memory transport. This works but sacrifices performance — P2P DMA is significantly faster than bouncing through system memory.
  4. User inquiry ([msg 6238]): The user asks if there are BIOS or kernel settings to restore P2P functionality. This is a reasonable question — many server BIOSes have options like "IOMMU Coverage" or "ACS Enable" that affect how the IOMMU handles peer-to-peer traffic.
  5. Systematic probe ([msg 6239] through [msg 6243]): The assistant checks the kernel command line, dmesg for IOMMU configuration, IOMMU features, IOMMU group assignments, and finally the NVIDIA driver parameters. Each step narrows the search space. The subject message is the final step in this probe. It checks the last remaining software-accessible knobs before the assistant must conclude that no OS-level or driver-level fix exists — the issue is baked into the hardware virtualization configuration.

Assumptions and Limitations

The assistant makes several assumptions in this message:

Output Knowledge Created

This message produces three pieces of knowledge:

  1. All four GPUs are in DMA-FQ IOMMU groups: Confirms uniform IOMMU translation mode across all GPUs. No GPU has a special passthrough exemption.
  2. No NVIDIA driver-level P2P workaround is active: Both NVreg_EnablePCIeRelaxedOrderingMode and NVreg_RegistryDwords are empty, meaning no driver-level tweaks have been applied and no known parameter exists to enable P2P through the IOMMU.
  3. The diagnostic path is exhausted: With no BIOS-visible IOMMU coverage settings (the user asked about this) and no driver knobs, the only remaining options are at the hardware virtualization level — either reconfigure the IOMMU to passthrough mode (breaking SEV-SNP isolation) or accept the NCCL P2P disable workaround.

The Broader Significance

This message is a textbook example of systematic debugging in complex systems. The assistant starts from a concrete failure (NCCL hang), traces it to corrupted P2P DMA, confirms the IOMMU as the culprit via IO_PAGE_FAULTs, applies a working workaround, and then — at the user's prompting — exhaustively checks whether a proper fix exists. The subject message is the final probe that returns negative results, allowing the team to make an informed decision: live with the workaround or reconfigure the virtualization layer.

For the ML deployment at hand, the answer is clear. The NCCL P2P disable workaround costs some performance — NCCL falls back to shared memory, which adds a copy through system RAM for each all-reduce operation — but the benchmarks show the system still achieves 2,800 tok/s at C=128 concurrency. The trade-off is acceptable, and the team moves on to other optimizations.

In the end, message [msg 6243] is a quiet but important moment: the confirmation that some problems cannot be solved with software alone, and that knowing when to stop digging is as valuable as knowing where to start.