The Moment the IOMMU Groups Refused to Merge: A Turning Point in the P2P DMA Investigation
Introduction
In any complex systems engineering effort, there comes a moment when a hypothesis meets reality and the outcome is definitive. Message 411 of this opencode session captures exactly such a moment. The user, having rebooted their ASUS ESC8000A-E13 server after disabling ACS (Access Control Services) in the BIOS, runs a battery of validation commands on the Proxmox host and reports the results back to the assistant. The data is unambiguous: despite the BIOS change, each of the eight NVIDIA RTX PRO 6000 Blackwell GPUs remains isolated in its own IOMMU group. The ACS disable did not merge them. The VM refuses to start with a cryptic error about IOMMU group mismatches. And perhaps most devastatingly, the PCIe link speeds confirm Gen5 x16 across all GPUs—the hardware is working perfectly, which only makes the P2P limitation more frustrating.
This message is not merely a status report. It is the experimental result that falsifies a central hypothesis of the preceding investigation. It forces a fundamental re-evaluation of what is possible in this virtualized environment and marks the transition from "can we fix this?" to "can we work around this?" Understanding this message requires understanding the chain of reasoning that led to it, the assumptions that were tested, and the new knowledge it creates for both the user and the assistant.
The Full Text of the Message
root@kpro6:~# for d in /sys/kernel/iommu_groups/*/devices/*; do
n=$(echo $d | cut -d/ -f5)
dev=$(lspci -nns ${d##*/} 2>/dev/null)
echo "$dev" | grep -qi nvidia && echo "IOMMU Group $n: $dev"
done
IOMMU Group 21: 61:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 25: 01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 2: 71:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 47: 11:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 51: f1:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 70: e1:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 74: 81:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
IOMMU Group 93: 91:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)
root@kpro6:~# cat /proc/cmdline | grep -o 'iommu=[^ ]*'
# Should show: iommu=pt
dmesg | grep -i 'iommu\|AMD-Vi' | grep -i 'mode\|passthrough\|translat' | head -5
iommu=on
iommu=pt
[ 1.145858] iommu: Default domain type: Passthrough (set via kernel command line)
root@kpro6:~# for gpu in 01:00.0 11:00.0 61:00.0 71:00.0 81:00.0 91:00.0 e1:00.0 f1:00.0; do
speed=$(lspci -vvv -s $gpu 2>/dev/null | grep "LnkSta:" | head -1)
echo "GPU $gpu: $speed"
done
GPU 01:00.0: LnkSta: Speed 32GT/s, Width x16
GPU 11:00.0: LnkSta: Speed 32GT/s, Width x16
GPU 61:00.0: LnkSta: Speed 32GT/s, Width x16
GPU 71:00.0: LnkSta: Speed 32GT/s, Width x16
GPU 81:00.0: LnkSta: Speed 32GT/s, Width x16
GPU 91:00.0: LnkSta: Speed 32GT/s, Width x16
GPU e1:00.0: LnkSta: Speed 32GT/s, Width x16
GPU f1:00.0: LnkSta: Speed 32GT/s, Width x16
root@kpro6:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq 2>/dev/null
# Check ASPM status
lspci -vvv -s 01:00.0 2>/dev/null | grep -i "ASPM\|LnkCtl:"
# Check NUMA topology
numactl --hardware
performance
3000000
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
-bash: numactl: command not found
root@kpro6:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 0 size: 257544 MB
node 0 free: 246610 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
node 1 size: 257967 MB
node 1 free: 256398 MB
node distances:
node 0 1
0: 10 32
1: 32 10
root@kpro6:~# lspci -k -s 01:00.0 | grep -i driver
root@kpro6:~# qm start 128
generating cloud-init ISO
PCI device mapping invalid (hardware probably changed): 'iommugroup' does not match for 'pro6000' (51 != 72) Why This Message Was Written: The Context and Motivation
To understand why this message exists, one must trace the investigation that preceded it. The conversation up to this point had been a multi-session odyssey of deploying the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs in a Proxmox virtualized environment. The team had successfully resolved NaN crashes during inference by selecting working attention backends, established baseline throughput benchmarks, and identified a critical performance bottleneck: cross-GPU communication latency was stuck at approximately 13 microseconds, far above what bare-metal deployments achieve.
The root cause was traced to the fact that Peer-to-Peer (P2P) DMA between GPUs was not functional inside the VM. The nvidia-smi topo -p2p command showed NS (Not Supported) for all GPU pairs, meaning that GPU-to-GPU data transfers had to go through host memory via the IOMMU, adding significant latency. This was especially painful for tensor-parallel inference, where model layers are split across GPUs and intermediate results must be exchanged frequently.
The assistant had developed a theory: ACS (Access Control Services), a PCIe capability that enforces isolation between devices on the same bridge or switch, was preventing the IOMMU from grouping GPUs together. If ACS could be disabled, the GPUs might merge into fewer IOMMU groups, and VFIO would then allow direct P2P DMA within those merged groups. This theory was articulated in message 408, where the assistant explained: "When ACS is disabled, PCIe bridges can forward P2P transactions directly between downstream devices. Devices that share a bridge/root complex get grouped into the same IOMMU group. Since they're in the same IOMMU group, VFIO treats them as one unit — and P2P DMA is allowed."
The user, who had been following this reasoning, asked in message 403: "pcie acs - wouldn't disabling permit p2p? or is it more hw level thing?" The assistant confirmed the theory and recommended disabling ACS in the BIOS under AMD CBS / NBIO common options → ACS → Disabled. The user rebooted the host to apply this change.
Message 411 is the user's report of what happened after that reboot. It is the experimental data that tests the ACS hypothesis. The motivation behind writing it is straightforward: the user is executing the validation commands the assistant prescribed in message 410, and relaying the raw output so the assistant can interpret whether the ACS change took effect.## The Experimental Result: What the Data Actually Shows
The message contains five distinct validation commands, each probing a different layer of the system. The first and most critical command iterates over all IOMMU groups on the host and prints only those containing NVIDIA devices. The output is devastatingly clear: eight GPUs, eight distinct IOMMU groups (21, 25, 2, 47, 51, 70, 74, 93). Not a single merge occurred. Before the ACS disable, the GPUs were in groups 42, 61, 28, 10, 101, 117, 90, and 72 (as noted in the assistant's earlier messages). The group numbers have changed—evidence that the BIOS re-enumeration did happen—but the isolation pattern is identical. Each GPU remains alone in its own group.
The second command checks the kernel command line and dmesg for IOMMU configuration. The output confirms iommu=on and iommu=pt (passthrough mode), with the kernel explicitly stating "Default domain type: Passthrough (set via kernel command line)." This is good news—the IOMMU is working correctly and in the most permissive mode available. But it also means the IOMMU is not the bottleneck; the grouping is determined by hardware topology, not software policy.
The third command checks PCIe link status. Every GPU reports Speed 32GT/s, Width x16—PCIe Gen5 at full width. This is the best possible physical connection. The GPUs are not bandwidth-limited by their links. The hardware is performing exactly as specified.
The fourth command checks CPU frequency governor (performance mode at 3GHz), ASPM status (disabled, as desired), and NUMA topology. The NUMA output reveals a dual-socket system with 128 CPUs spread across two NUMA nodes, each with approximately 256 GB of memory. The node distances (10 for local, 32 for remote) confirm the expected AMD EPYC topology.
The fifth command attempts to start the VM and fails with: PCI device mapping invalid (hardware probably changed): 'iommugroup' does not match for 'pro6000' (51 != 72). This is a Proxmox-specific error. The PCI mapping file (/etc/pve/mapping/pci.cfg) still contains the old IOMMU group numbers from before the BIOS re-enumeration. Since the groups changed (e.g., the GPU formerly in group 72 is now in group 51), Proxmox refuses to start the VM with stale mappings. This is a practical consequence of the BIOS change: even though ACS disable didn't merge the groups, it did renumber them, breaking the VM configuration.
The Assumptions That Were Tested and Falsified
This message is valuable precisely because it tests several interconnected assumptions, and the results force a reassessment of each one.
Assumption 1: ACS disable would merge IOMMU groups. This was the central hypothesis. The assistant believed that disabling ACS in the BIOS would cause GPUs sharing a PCIe bridge or root complex to be placed in the same IOMMU group. The data shows this did not happen. The GPUs remain in separate groups, proving that ACS was not the mechanism enforcing their isolation.
Assumption 2: The GPUs share PCIe bridges or switches. The ACS theory implicitly assumed that there was some PCIe topology element—a bridge, a switch, a root port—that connected multiple GPUs and where ACS could enforce isolation. The experimental result disproves this. The fact that ACS disable had no effect on grouping strongly suggests that each GPU is on its own dedicated PCIe root complex with no shared bridge between any pair. On the AMD EPYC platform, each GPU slot is connected to its own root port on the CPU's internal PCIe controller, and these root ports are independent of each other. ACS operates within a single PCIe hierarchy (downstream of a bridge or root port), but it cannot merge devices that are on entirely separate hierarchies.
Assumption 3: The BIOS ACS setting was correctly applied. There is always the possibility that the BIOS option was misconfigured, not actually present, or overridden by another setting. However, the fact that IOMMU group numbers changed (from the old set to the new set) indicates that the BIOS did re-enumerate the PCIe topology. The group renumbering is itself evidence that something changed. The most parsimonious explanation is that ACS was indeed disabled, but it had no effect on grouping because there was nothing to merge.
Assumption 4: The VM would start normally after the BIOS change. The user likely expected that after rebooting with ACS disabled, they could simply start the VM and test P2P inside. Instead, they encountered the Proxmox mapping error. This is a secondary but important failure: the BIOS re-enumeration invalidated the PCI device mapping file, creating an operational blocker even before the P2P question could be tested.
Input Knowledge Required to Understand This Message
A reader needs substantial context to make sense of this message. First, they need to understand what IOMMU groups are and why they matter for GPU virtualization. The IOMMU (Input/Output Memory Management Unit) is a hardware component that translates device DMA addresses to physical memory addresses. Devices in the same IOMMU group share a translation domain, meaning they can DMA to each other's memory without trapping through the hypervisor. Devices in different IOMMU groups are isolated—VFIO must mediate any cross-group DMA, which adds latency. This is the fundamental reason P2P performance suffers in virtualized environments.
Second, the reader needs to understand ACS (Access Control Services). ACS is a PCIe capability defined in the PCIe Base Specification that allows a bridge or switch to control whether peer-to-peer transactions are forwarded between downstream ports or redirected upstream. VFIO uses ACS to enforce IOMMU group boundaries: if ACS is enabled on a bridge, devices downstream of different ports are placed in separate groups for security isolation. The theory was that disabling ACS would cause VFIO to merge these groups.
Third, the reader needs to understand the hardware topology of the ASUS ESC8000A-E13 motherboard. This is an AMD EPYC platform with eight GPU slots, each connected to a dedicated PCIe root complex on the CPU. There is no PCIe switch that fans out to multiple GPUs. This topology is designed for maximum per-GPU bandwidth (each GPU gets a direct Gen5 x16 link to the CPU), but it inherently prevents IOMMU group merging because there is no shared PCIe element between any two GPUs.
Fourth, the reader needs to understand Proxmox PCI mapping. The file /etc/pve/mapping/pci.cfg maps PCI devices to IOMMU groups for VM passthrough. When the BIOS re-enumerates PCIe devices (which happens after any PCIe-related BIOS change), the IOMMU group numbers can change, requiring the mapping file to be updated. The error message 'iommugroup' does not match for 'pro6000' (51 != 72) indicates that the mapping file still references the old group number 72 for a device now in group 51.
Output Knowledge Created by This Message
This message creates several pieces of new knowledge that advance the investigation.
Knowledge 1: ACS disable is ineffective for this hardware topology. The primary experimental result is that disabling ACS does not merge IOMMU groups on this platform. This is not a failure of the ACS mechanism per se—it is a consequence of the physical topology where each GPU is on its own root complex. ACS only controls forwarding within a PCIe hierarchy, and there is no shared hierarchy between GPUs. This knowledge is definitive: no amount of BIOS or kernel tweaking can create a shared PCIe bridge where none exists in hardware.
Knowledge 2: The IOMMU group numbers have changed. The new group numbers (21, 25, 2, 47, 51, 70, 74, 93) differ from the old ones (42, 61, 28, 10, 101, 117, 90, 72). This means the PCI mapping file must be updated before the VM can start. The user now has a concrete operational task: fix the mapping file.
Knowledge 3: The PCIe links are confirmed at Gen5 x16 with ASPM disabled. This is positive validation that the physical layer is optimal. The GPUs have maximum bandwidth and no power management overhead. Any remaining performance issues are not due to PCIe link configuration.
Knowledge 4: IOMMU passthrough mode is active. The kernel confirms "Default domain type: Passthrough." This is the most permissive IOMMU mode, which minimizes translation overhead for DMA. It rules out the possibility that IOMMU translation mode is causing the latency.
Knowledge 5: The NUMA topology is healthy. Two NUMA nodes with 256 GB each, node distance 32 for remote access. This is standard for dual-socket AMD EPYC. The CPU frequency governor is set to performance mode at 3 GHz. These are all favorable conditions for GPU workloads.
Knowledge 6: The VM boot is blocked by a stale mapping. The immediate next step is clear: update /etc/pve/mapping/pci.cfg with the new IOMMU group numbers. This is a mechanical fix but a necessary prerequisite to any further P2P testing.## The Thinking Process Visible in the Message
Although this message is primarily raw command output from the user, the structure of the commands reveals a deliberate diagnostic strategy. The user did not simply run one command and report the result. They executed a systematic battery of five probes, each targeting a different subsystem:
- IOMMU group enumeration (the core hypothesis test)
- Kernel IOMMU configuration (verifying the foundation)
- PCIe link status (checking physical layer health)
- Power management and NUMA topology (checking performance settings)
- VM startup (the practical consequence) This ordering is not accidental. The user starts with the most critical question—did the IOMMU groups merge?—and works outward to confirm that all supporting subsystems are healthy. The inclusion of the
numactl --hardwareoutput (even though it required a second invocation after the initial "command not found") shows attention to completeness. The user is not just dumping data; they are methodically checking each layer that the assistant's diagnostic plan (message 410) prescribed. The message also reveals a certain pragmatism. The user includes the shell comments (# Should show: iommu=pt,# Check ASPM status) from the assistant's instructions, suggesting they are working through a checklist. This is not a haphazard exploration but a disciplined execution of a diagnostic protocol. The final command—qm start 128—is the moment of truth. The user has gathered all the diagnostic data and now attempts to bring the VM online to test P2P inside. The failure is immediate and specific: the PCI mapping is invalid. The error message is included verbatim, giving the assistant all the information needed to diagnose and fix the problem.
Mistakes and Incorrect Assumptions
The most significant incorrect assumption in this message is not made by the user but by the assistant in the preceding messages. The ACS theory was reasonable—it is well-documented that ACS controls IOMMU group assignment in VFIO—but it was applied to a hardware topology where ACS had no leverage. The assistant assumed that the GPUs shared some PCIe topology element where ACS could enforce isolation. In reality, each GPU is on its own root complex, and ACS has nothing to merge.
This is a subtle mistake that is easy to make. The assistant had previously observed that nvidia-smi topo -m showed all GPUs as PHB (Peer-to-Peer via Host Bridge), which is the topology classification for devices on different root complexes but the same CPU socket. The assistant interpreted this as evidence that ACS might be the blocking factor, but the PHB classification actually already told the true story: the GPUs are on separate root complexes and communicate through the host bridge (the CPU's data fabric). ACS cannot merge what was never connected.
A secondary mistake was the assumption that the BIOS ACS setting would be straightforward to apply and would have an immediate observable effect. In practice, BIOS changes on server hardware can be complex—the setting might be named differently, hidden behind multiple menus, or have dependencies on other settings. The fact that IOMMU groups renumbered but did not merge suggests the BIOS change was applied but was structurally irrelevant.
The user's own assumption—that disabling ACS would permit P2P—was based on the assistant's recommendation and was entirely reasonable given the information available at the time. The user asked the right question in message 403 ("wouldn't disabling permit p2p? or is it more hw level thing?"), and the assistant confirmed the theory. Neither party can be faulted for testing a hypothesis that turned out to be incorrect; that is the nature of diagnostic engineering.
The Deeper Significance: A Fundamental Hardware Constraint
This message marks a critical inflection point in the investigation. Before this message, there was hope that a software or BIOS configuration change could enable P2P DMA in the VM. After this message, that hope is extinguished. The eight GPUs are on eight separate PCIe root complexes, and no amount of ACS toggling, kernel parameter tweaking, or VFIO configuration can create a shared PCIe hierarchy where none exists in hardware.
This is a fundamental constraint of the ASUS ESC8000A-E13 motherboard design. The platform is optimized for maximum per-GPU bandwidth—each slot gets a dedicated Gen5 x16 link directly to the CPU—at the cost of P2P topology. In bare metal, the AMD EPYC Data Fabric (Infinity Fabric) can route P2P traffic between root complexes internally, which is why bare-metal systems show NODE or PHB topology with working P2P. But in a virtualized environment, VFIO cannot leverage the Data Fabric for cross-group DMA because the IOMMU enforces isolation between groups.
The practical implication is stark: for this specific hardware in this specific virtualization setup, cross-GPU communication will always incur the ~13 microsecond latency penalty of staged transfers through host memory. The investigation must pivot from "how do we enable P2P?" to "how do we minimize the impact of its absence?" This might involve optimizing NCCL communication patterns, reducing the frequency of cross-GPU transfers through model architecture changes, or accepting the latency and tuning batch sizes to compensate.
The Path Forward: What This Message Enables
Despite being a negative result, this message creates a clear path forward. The immediate action item is to update the Proxmox PCI mapping file so the VM can start. The user now knows the new IOMMU group numbers (21, 25, 2, 47, 51, 70, 74, 93) and can map them correctly. Once the VM is running, the team can verify the P2P status inside (almost certainly still NS) and begin exploring alternative optimization strategies.
The message also provides a comprehensive health check of the host system. All eight GPUs are detected at Gen5 x16 with ASPM disabled. The IOMMU is in passthrough mode. The CPU is in performance governor at 3 GHz. The NUMA topology is clean with two nodes and 512 GB total memory. These are all green lights. The only red light is the P2P limitation, which is now understood to be a hardware topology constraint rather than a configuration error.
For the broader project of deploying GLM-5-NVFP4 across eight GPUs with SGLang, this means the team must optimize within the constraint of no P2P. Strategies might include: using NVLink if the GPUs support it (the RTX PRO 6000 Blackwell may have NVLink, but it would require bridge connections that may not be present in this setup), optimizing NCCL to use shared memory (SHM) paths efficiently, tuning the tensor parallelism to minimize cross-GPU communication, or accepting the latency and compensating with larger batch sizes and more concurrent requests.
Conclusion
Message 411 is a masterclass in diagnostic data presentation. In a few dozen lines of shell output, it communicates the result of a critical hypothesis test, validates the health of supporting subsystems, reveals an unexpected operational blocker, and provides all the information needed to plan the next steps. The message is honest about its negative result—ACS disable did not merge the IOMMU groups—but it is also rich with positive validation: Gen5 x16 links, passthrough IOMMU mode, performance governor, clean NUMA topology.
The message teaches an important lesson about systems engineering: sometimes the most valuable experimental result is a clear falsification. The ACS hypothesis was elegant and plausible, but it was wrong for this hardware. The data proved it wrong definitively, saving countless hours of further fruitless BIOS tweaking. The investigation can now move forward with a correct understanding of the constraint: the hardware topology itself prevents P2P in this virtualized environment, and the solution must lie elsewhere.
For anyone working with GPU virtualization on AMD EPYC platforms, this message serves as a cautionary tale. Before attempting to enable P2P in a VM, verify the PCIe topology. If each GPU is on its own root complex, ACS manipulation will not help. The only paths to P2P in such a topology are either bare-metal deployment, specialized hardware with PCIe switches, or NVLink bridges between GPUs. In the absence of these, the optimization effort must focus on working efficiently within the P2P-less reality.
The user, by methodically executing this diagnostic protocol and reporting the results with precision, has advanced the investigation more than any speculative discussion could have. This message is the turning point where theory meets reality, and reality wins.