The Topology Revelation: Mapping Physical NUMA to Virtual GPUs in a Proxmox ML Environment

Introduction

In the complex dance of deploying large language models across multiple GPUs in a virtualized environment, few moments are as clarifying as the one captured in message 384 of this opencode session. The assistant, having spent several rounds investigating PCIe topology, IOMMU groups, and BAR allocation failures, finally receives the critical piece of data that connects the physical hardware layout to the virtual machine's GPU configuration. This message represents a turning point—a moment of synthesis where scattered observations about NUMA nodes, PCIe bus addresses, and Q35 chipset behavior coalesce into a coherent understanding of the machine's fundamental topology.

The message is deceptively short, but it packs a tremendous amount of analytical weight. It begins with a table that maps the Proxmox hostpci device assignments to their physical PCI addresses and host NUMA nodes, revealing a crucial insight: the VM's GPUs are assigned in reverse NUMA order, with GPU0-3 physically residing on NUMA node 1 and GPU4-7 on NUMA node 0. This discovery sets off a chain of reasoning about how to properly expose NUMA topology to the VM, ultimately leading to a pragmatic pivot from hardware reconfiguration to software-based topology overrides.

The Message in Full

The assistant's message reads:

Now I have the full mapping. The pro6000 mapping lists 8 GPUs in order, and hostpci0-7 are assigned in that order:

| hostpci | VM Bus | Physical PCI | Host NUMA | |---------|--------|-------------|-----------| | hostpci0 | 01:00.0 | f1:00.0 | NUMA 1 | | hostpci1 | 02:00.0 | e1:00.0 | NUMA 1 | | hostpci2 | 03:00.0 | 91:00.0 | NUMA 1 | | hostpci3 | 04:00.0 | 81:00.0 | NUMA 1 | | hostpci4 | 05:00.0 | 71:00.0 | NUMA 0 | | hostpci5 | 06:00.0 | 61:00.0 | NUMA 0 | | hostpci6 | 07:00.0 | 11:00.0 | NUMA 0 | | hostpci7 | 08:00.0 | 01:00.0 | NUMA 0 |

So: VM GPUs 0-3 are on physical NUMA 1, VM GPUs 4-7 are on physical NUMA 0. On bare metal the topology would be:

- GPU0-3: NODE to each other, SYS to GPU4-7 - GPU4-7: NODE to each other, SYS to GPU0-3

But in the VM all 8 show PHB with no NUMA distinction. To fix this properly, we need to reorder the mapping so NUMA-local GPUs are grouped on the same Q35 PCIe controller, AND we need the VM to expose proper NUMA affinity.

Currently in Q35, GPUs on 00:10.x (buses 01-04, hostpci0-3) are on one PCIe root port controller, and GPUs on 00:1c.x (buses 05-08, hostpci4-7) are on a different one. The mapping already groups NUMA 1 GPUs on 00:10.x and NUMA 0 on 00:1c.x — so the physical NUMA grouping coincidentally aligns with the two Q35 PCIe controller groups. Good.

The problem is the VM doesn't expose this as separate NUMA nodes. To fix this, we need to use QEMU's pcie-expander-bus with NUMA node assignments. However, Proxmox doesn't natively support pcie-expander-bus through its config syntax — we'd need custom -device args, which conflicts with Proxmox's automatic hostpci device creation.

There's a simpler approach that might work: use an NCCL topology XML file inside the VM to tell NCCL the real topology, regardless of what nvidia-smi topo shows.

Let me create that: [todowrite] {"todos":[{"content":"Investigate Proxmox host IOMMU/P2P configuration","priority":"high","status":"completed"},{"content":"Enable IOMMU on Proxmox host kernel cmdline","priority":"high","status":"completed"},{"content":"Fix BAR allocation for 8 GPUs on Q35 + pcie=1","priority":"high","status":"completed"},{"content":"Fix GPU NUMA topology in VM to match bare metal","priority":"high","status":"in_progress"}...]}

Why This Message Was Written: The Reasoning and Motivation

This message is the culmination of an investigative thread that began several rounds earlier. The user and assistant had been wrestling with a persistent performance problem: cross-GPU communication in the Proxmox VM was bottlenecked at approximately 13 microseconds per small transfer, with bandwidth topping out around 41 GB/s for large transfers. This was far below what the eight RTX PRO 6000 Blackwell GPUs, connected via PCIe Gen5 x16 links, should have been capable of.

The investigation had already taken several turns. The assistant had helped migrate the VM from the legacy i440FX chipset to Q35 with proper PCIe passthrough (pcie=1), fixed a BAR allocation failure that prevented 6 of 8 GPUs from being detected, and attempted to disable ACS (Access Control Services) to merge IOMMU groups. Each step peeled back another layer of the problem, only to reveal a deeper constraint.

The critical context for this message is the user's query in [msg 370], where they pasted a nvidia-smi topo output showing the bare-metal topology with NODE and SYS connections. The assistant had been working with incomplete information about how the physical GPUs mapped to the VM's GPU indices. Without knowing which physical PCI address corresponded to which VM GPU, it was impossible to reason about NUMA affinity or propose targeted fixes.

The user's execution of cat /etc/pve/mapping/pci.cfg in [msg 383] provided the missing link. This file contained the pro6000 resource mapping that Proxmox uses to assign physical PCI devices to hostpci slots. The assistant's message is the immediate synthesis of this new data.

The motivation is clear: the assistant needs to understand the physical-to-virtual mapping to determine whether the VM's PCIe topology can be made to match the bare-metal topology, and if so, how. This is not an abstract exercise—it has direct performance implications. On bare metal, NCCL (NVIDIA Collective Communications Library) uses the NODE vs SYS distinction to optimize communication algorithms. Intra-NUMA transfers (NODE) are faster than cross-NUMA transfers (SYS), and NCCL can schedule communication patterns to prefer local transfers. In the VM, all GPUs show PHB (PCIe Host Bridge), which means NCCL treats every GPU pair identically, losing the opportunity for topology-aware optimization.

How Decisions Were Made

This message contains several implicit and explicit decisions, each grounded in the assistant's analysis of the new data.

Decision 1: The mapping table. The assistant constructs a four-column table mapping hostpci index → VM bus address → physical PCI address → host NUMA node. This is the foundational analytical act of the message. The assistant had previously gathered the NUMA node assignments for each physical GPU address in [msg 372] (user ran the command on the host). The mapping config from /etc/pve/mapping/pci.cfg provides the order in which physical GPUs are assigned to hostpci slots. By cross-referencing these two data sources, the assistant can determine the NUMA node of each VM GPU.

The table reveals an interesting pattern: VM GPUs 0-3 (hostpci0-3) map to physical addresses f1:00.0, e1:00.0, 91:00.0, and 81:00.0, all on NUMA node 1. VM GPUs 4-7 (hostpci4-7) map to 71:00.0, 61:00.0, 11:00.0, and 01:00.0, all on NUMA node 0. This is actually the reverse of what one might expect—the first four GPUs in the VM are on the second NUMA node.

Decision 2: Assessing the existing alignment. The assistant then checks whether the Q35 chipset's PCIe controller grouping already aligns with the NUMA grouping. In Q35, hostpci0-3 are placed on the 00:10.x PCIe root port controller, while hostpci4-7 are on 00:1c.x. Since hostpci0-3 happen to be NUMA 1 GPUs and hostpci4-7 are NUMA 0 GPUs, the physical NUMA grouping coincidentally aligns with the two Q35 PCIe controller groups. The assistant notes this with "Good."—it's a fortunate accident that the mapping order happens to group NUMA-local GPUs on the same virtual PCIe controller.

Decision 3: Rejecting the hardware approach. The assistant considers using QEMU's pcie-expander-bus with NUMA node assignments to properly expose the NUMA topology to the VM. This would involve custom -device arguments passed via qm set 128 -args '...'. However, the assistant correctly identifies a conflict: Proxmox's hostpci syntax automatically creates the virtual PCIe devices, and custom -device args would interfere with this automatic creation. The decision to reject this approach is pragmatic—it recognizes the limitations of working within Proxmox's configuration framework.

Decision 4: Pivoting to NCCL topology XML. The assistant proposes a simpler, software-based approach: creating an NCCL topology XML file inside the VM that tells NCCL the real GPU topology, overriding what nvidia-smi topo reports. This is a clever workaround that operates at the application layer rather than the virtualization layer. NCCL supports a NCCL_TOPO_FILE environment variable that points to an XML file describing the GPU interconnect topology. By creating a file that mirrors the bare-metal topology (NODE within each NUMA group, SYS across groups), NCCL can make optimal communication decisions even though the VM's virtual PCIe topology doesn't expose this information.

This decision represents a significant strategic pivot. The assistant had been pursuing hardware-level fixes—IOMMU passthrough, ACS disable, Q35 chipset migration, BAR reallocation. Each of these was aimed at making the VM's hardware topology more closely match bare metal. The NCCL topology file approach is a software-level fix that works around the hardware limitations rather than trying to fix them. It's a recognition that some virtualization constraints are fundamental and cannot be overcome by reconfiguration alone.

Assumptions Made by the Assistant

Several assumptions underpin this message, some explicit and some implicit.

Assumption 1: The mapping order is deterministic. The assistant assumes that hostpci0 gets the first entry in the pro6000 mapping, hostpci1 gets the second, and so on. This is consistent with Proxmox's documented behavior for resource mappings, but the assistant doesn't verify it empirically. The assumption is reasonable given Proxmox's design, but a verification step (e.g., checking which physical GPU appears at which VM bus address) would have been more rigorous.

Assumption 2: The Q35 PCIe controller grouping is meaningful for performance. The assistant assumes that having GPUs on separate Q35 PCIe root port controllers (00:10.x vs 00:1c.x) provides some topology benefit. In practice, the Q35 chipset's PCIe root ports are all virtual constructs within QEMU—they don't correspond to physical PCIe switches or separate buses in the host. The actual data path for inter-GPU communication still goes through QEMU's virtual device model, which stages data through host memory. The 00:10.x vs 00:1c.x distinction may not have any real performance impact.

Assumption 3: NCCL respects the topology file. The assistant assumes that NCCL will honor the NCCL_TOPO_FILE environment variable and use the provided topology for communication optimization. This is generally true, but NCCL's behavior depends on the version and the specific algorithms being used. Some NCCL versions may fall back to hardware-reported topology for certain operations.

Assumption 4: The NCCL topology file approach doesn't have side effects. The assistant implicitly assumes that overriding the topology won't cause NCCL to make incorrect decisions (e.g., attempting P2P transfers that aren't actually possible in the VM). This is a potential risk—if NCCL believes GPUs are connected via NODE (same NUMA node, different PCIe host bridges), it might attempt direct peer-to-peer transfers that fail in the virtualized environment.

Mistakes or Incorrect Assumptions

The most significant potential error in this message is the implicit assumption that the NCCL topology file will meaningfully improve performance. While NCCL will use the provided topology for scheduling decisions, the actual data path is still constrained by QEMU's virtual device model. The 13-microsecond latency floor for small transfers is a fundamental property of the GPU→host memory→GPU staged copy path, not something NCCL can optimize away. Even if NCCL schedules intra-NUMA transfers preferentially, those transfers still go through the same QEMU-mediated path.

The assistant also may be overestimating the importance of the NODE vs SYS distinction for NCCL's performance. On bare metal, the difference between intra-NUMA and cross-NUMA transfers is meaningful because they traverse different physical interconnects (Infinity Fabric within a socket vs Infinity Fabric between sockets). In the VM, both paths are identical—they both go through QEMU's virtual PCIe to host memory. The NCCL topology file might cause NCCL to prefer certain GPU pairs for communication, but the actual bandwidth and latency would be the same regardless of which pair is used.

Additionally, the assistant's conclusion that "the mapping already groups NUMA 1 GPUs on 00:10.x and NUMA 0 on 00:1c.x — so the physical NUMA grouping coincidentally aligns" is somewhat misleading. The alignment is coincidental only in the sense that the mapping order happened to place NUMA 1 GPUs first. But the Q35 chipset's two PCIe root port controllers are virtual constructs—they don't correspond to physical NUMA nodes. The alignment doesn't provide any performance benefit in the current configuration because the VM doesn't expose NUMA node information to the guest OS.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several domains:

Proxmox VE configuration: Understanding how hostpci device assignment works, what resource mappings (/etc/pve/mapping/pci.cfg) are, and how Proxmox assigns physical PCI devices to virtual machines. The reader must know that hostpci0-7 are sequential assignments that map to specific virtual PCIe buses in the VM.

PCIe topology and NUMA architecture: Understanding PCIe bus addresses (e.g., 0000:f1:00.0), NUMA nodes, and how GPUs are connected to CPU sockets via PCIe root complexes. The reader needs to know that on a dual-socket AMD EPYC system, GPUs attached to different CPU sockets are on different NUMA nodes, and that inter-socket communication traverses the Infinity Fabric interconnect.

QEMU/Q35 chipset internals: Understanding how QEMU's Q35 chipset emulates PCIe root ports, how pcie-expander-bus works, and how virtual PCIe topology is constructed. The reader needs to know that Q35 provides multiple PCIe root port controllers (00:10.x and 00:1c.x) that can host PCIe devices.

NCCL internals: Understanding how NCCL uses topology information for communication optimization, what NODE, SYS, PHB, PIX mean in nvidia-smi topo output, and how NCCL_TOPO_FILE works. The reader needs to know that NCCL builds communication algorithms (ring, tree, etc.) based on the perceived GPU interconnect topology.

The broader context of the session: The reader needs to know that this is part of an effort to deploy the GLM-5-NVFP4 model on 8 RTX PRO 6000 Blackwell GPUs, that the system is a Proxmox VM on an ASUS ESC8000A-E13 motherboard with dual AMD EPYC processors, and that previous attempts to enable P2P DMA have been blocked by the hardware topology (each GPU on its own PCIe root complex).

Output Knowledge Created

This message creates several important pieces of knowledge:

The physical-to-virtual GPU mapping: The most concrete output is the mapping table showing which physical GPU (by PCI address and NUMA node) corresponds to which VM GPU index. This is essential information for any subsequent topology optimization.

The NUMA alignment assessment: The message establishes that the existing hostpci mapping order happens to group NUMA-local GPUs on the same Q35 PCIe controller, providing a foundation for further optimization.

The feasibility analysis of pcie-expander-bus: The message documents why the pcie-expander-bus approach is impractical with Proxmox's configuration framework, saving future effort on this dead end.

The NCCL topology file strategy: The most forward-looking output is the proposal to use an NCCL topology XML file as a software-level workaround. This creates a new avenue for investigation that the assistant can pursue in subsequent messages.

The todo list update: The message updates the task tracking, marking IOMMU investigation, kernel cmdline changes, and BAR allocation as completed, and marking GPU NUMA topology as in-progress. This provides structure to the ongoing investigation.

The Thinking Process Visible in Reasoning

The message reveals a structured analytical process. The assistant:

Receives raw data (the pci.cfg mapping)
Cross-references it with previously gathered NUMA node information
Constructs a synthesized view (the mapping table)
Draws conclusions about the NUMA alignment
Evaluates options (pcie-expander-bus vs NCCL topology file)
Selects a path forward (NCCL topology file)
Updates task tracking to reflect progress The reasoning is methodical and evidence-based. Each conclusion is supported by data from previous messages. The assistant doesn't jump to solutions—it first establishes the facts, then evaluates alternatives against the constraints of the environment. The message also shows the assistant's ability to recognize when a hardware-level fix is impractical and pivot to a software-level workaround. This is a mature engineering judgment: rather than continuing to fight against fundamental virtualization constraints, find a way to achieve the same effect through a different mechanism.

Conclusion

Message 384 is a pivotal moment in the opencode session—a synthesis of scattered data points into a coherent understanding of the VM's GPU topology. It demonstrates the importance of having complete information before making decisions, the value of cross-referencing data from multiple sources, and the art of knowing when to pivot from hardware fixes to software workarounds.

The message's legacy is the NCCL topology file strategy, which represents a pragmatic compromise between the ideal (full hardware P2P DMA) and the achievable (software-level topology hints). Whether this strategy ultimately improves performance remains to be seen in subsequent messages, but the analytical groundwork laid in this message is solid.

For anyone working with multi-GPU VMs on Proxmox, this message offers a valuable lesson: sometimes the most productive path forward is not to fight against virtualization constraints, but to work with them through application-layer configuration. The topology is not destiny—but understanding it is the first step to working around it.