The NUMA Revelation: A Pivotal Data Point in the Quest for GPU P2P in a Proxmox VM

Introduction

In the complex and often frustrating journey of configuring high-performance GPU inference workloads inside a virtualized environment, few moments are as decisive as the one captured in message 372 of this opencode session. At first glance, it appears to be nothing more than the output of a simple shell loop — a user running a for loop on a Proxmox host to check which NUMA node each GPU belongs to. The output is terse: four GPUs on NUMA 0, four on NUMA 1. Yet this single message represents a critical juncture in a multi-hour debugging saga, one that had already spanned kernel parameter modifications, chipset migrations, BIOS-level ACS manipulation, and deep dives into the PCIe topology of an AMD EPYC server. This article examines why this message was written, what knowledge it required and produced, and how it shaped the trajectory of the investigation.

The Context: A Desperate Search for P2P

To understand message 372, one must first appreciate the context that led to it. The session's overarching goal was to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang, a high-performance inference engine. Early benchmarks revealed a critical bottleneck: cross-GPU communication latency was stuck at approximately 13 microseconds per small transfer, and nvidia-smi topo -p2p reported "NS" (Not Supported) for all GPU pairs. For tensor-parallel inference workloads that require frequent all-reduce operations across model layers, this latency floor was a serious performance limiter.

The assistant and user had already attempted numerous remedies. They migrated the VM from the legacy i440FX chipset to Q35 with proper PCIe device passthrough. They added pci=realloc to the guest kernel to fix BAR allocation failures that initially prevented six of eight GPUs from being detected. On the Proxmox host, they enabled IOMMU passthrough (amd_iommu=on iommu=pt) and even attempted to disable Access Control Services (ACS) in the BIOS to merge IOMMU groups. None of these efforts enabled P2P.

The investigation had reached a sobering conclusion: the ASUS ESC8000A-E13 motherboard's design placed each GPU on its own dedicated PCIe root complex, with no shared PCIe switch. This hardware topology fundamentally prevented VFIO from granting direct P2P DMA access, regardless of software configuration. The ACS disable attempt had proven this conclusively — it renumbered the IOMMU groups but failed to merge them.

The Message Itself

Message 372 is a user message containing the verbatim output of a bash command executed on the Proxmox host (hostname kpro6):

root@kpro6:~# for gpu in 01:00.0 11:00.0 61:00.0 71:00.0 81:00.0 91:00.0 e1:00.0 f1:00.0; do
  numa=$(cat /sys/bus/pci/devices/0000:$gpu/numa_node 2>/dev/null)
  echo "GPU $gpu -> NUMA $numa"
done
GPU 01:00.0 -> NUMA 0
GPU 11:00.0 -> NUMA 0
GPU 61:00.0 -> NUMA 0
GPU 71:00.0 -> NUMA 0
GPU 81:00.0 -> NUMA 1
GPU 91:00.0 -> NUMA 1
GPU e1:00.0 -> NUMA 1
GPU f1:00.0 -> NUMA 1

The command iterates over eight PCIe device addresses — the physical locations of the eight GPUs on the host's PCIe bus — and reads each device's numa_node file from the Linux kernel's sysfs interface. The output reveals a perfect 4+4 split: the first four GPUs in the list reside on NUMA node 0, while the remaining four reside on NUMA node 1. This mirrors the dual-socket AMD EPYC architecture of the server, where each CPU socket corresponds to a NUMA domain.

Why This Message Was Written: The Reasoning and Motivation

The message was written in direct response to the assistant's request in the preceding message (index 371). The assistant had just completed a detailed comparison between the VM's reported GPU topology (all PHB — PCIe Host Bridge) and a bare-metal topology the user had pasted (showing NODE for intra-socket connections and SYS for cross-socket connections). The assistant identified two critical deficiencies in the VM's topology:

Missing NUMA affinity: The bare-metal topology showed GPUs 0-3 on NUMA node 0 and GPUs 4-7 on NUMA node 1, with corresponding CPU affinity masks. In the VM, all GPUs reported PHB with no NUMA distinction, meaning NCCL (NVIDIA Collective Communications Library) could not optimize communication based on locality.
Missing NODE vs SYS distinction: On bare metal, intra-socket GPU pairs showed NODE (lower latency, within the same NUMA node) while cross-socket pairs showed SYS (traversing the Infinity Fabric inter-socket link). In the VM, everything was uniformly PHB, preventing NCCL from building an optimal communication topology. The assistant hypothesized that by correctly assigning NUMA affinity in the VM configuration, the topology could be improved — potentially recovering the NODE/SYS distinction that NCCL uses to optimize collective operations. The command the assistant asked the user to run was designed to determine the precise mapping of physical GPU PCIe addresses to NUMA nodes, which would then inform how to configure the VM's PCIe topology using QEMU features like pcie-expander-bus. The motivation, therefore, was to gather the essential data needed to attempt a virtual PCIe topology reconstruction — a last-ditch effort to make the VM's GPU interconnect topology resemble bare metal as closely as possible, even if true hardware P2P remained unattainable.## Assumptions Embedded in the Message The message carries several implicit assumptions worth examining. First, the user assumed that the PCIe addresses provided by the assistant were correct and corresponded to the eight GPUs. This was a reasonable assumption — the assistant had derived these addresses from the host's PCIe topology earlier in the session — but it highlights the collaborative nature of the debugging process, where each party relies on the other's prior work. Second, the message assumes that reading /sys/bus/pci/devices/0000:$gpu/numa_node is the correct and authoritative way to determine NUMA affinity. This is indeed the standard Linux kernel interface for this information, but it's worth noting that the numa_node file may report -1 for devices whose NUMA node is unknown or unset. The fact that it returned clean 0 and 1 values confirms that the host kernel has proper NUMA topology awareness and that the GPUs are correctly assigned. Third, there is an implicit assumption that NUMA affinity information, once obtained, can be translated into actionable VM configuration changes. This assumption proved optimistic — as the subsequent conversation would reveal, constructing a virtual PCIe topology that mirrors the physical NUMA split is technically challenging with Proxmox's configuration model, often requiring raw QEMU command-line arguments that conflict with Proxmox's managed device assignment.

Input Knowledge Required

To fully understand this message, a reader needs several layers of context. At the hardware level, one must understand that the server has a dual-socket AMD EPYC CPU architecture, where each socket has its own memory controller and PCIe root complex, forming distinct NUMA (Non-Uniform Memory Access) domains. GPUs connected to different sockets will have different memory access latencies and different paths to each other.

At the Linux kernel level, one must know about the sysfs interface at /sys/bus/pci/devices/ and the numa_node file, which exposes the NUMA node assignment for each PCIe device. This is a standard interface used by tools like nvidia-smi and lstopo to report topology information.

At the virtualization level, one must understand that Proxmox VE uses QEMU/KVM as its hypervisor, and that PCIe passthrough (via VFIO) assigns physical PCIe devices to virtual machines. The mapping between physical PCIe addresses (like 01:00.0) and VM-visible PCIe addresses (like 0000:01:00.0 in the guest) is not necessarily one-to-one — the VM's PCIe topology is constructed by QEMU based on the VM configuration.

At the application level, one must understand that NCCL uses topology information (reported by nvidia-smi topo -m) to select optimal communication algorithms and paths. The distinction between NODE (same NUMA node, different PCIe host bridges), SYS (cross-NUMA via inter-socket interconnect), and PHB (generic PCIe host bridge) directly affects NCCL's performance.

Output Knowledge Created

The message created several valuable pieces of knowledge. Most directly, it confirmed that the eight GPUs are evenly split across the two NUMA nodes — four on NUMA 0 and four on NUMA 1. This matched the bare-metal topology the user had previously shared, which showed GPUs 0-3 on NUMA 0 and GPUs 4-7 on NUMA 1.

More subtly, the message revealed the specific PCIe addresses of the GPUs on each NUMA node. The addresses 01:00.0, 11:00.0, 61:00.0, and 71:00.0 are on NUMA 0, while 81:00.0, 91:00.0, e1:00.0, and f1:00.0 are on NUMA 1. This granular mapping is essential for any attempt to construct a virtual PCIe topology that preserves NUMA locality — for example, by placing each group of four GPUs behind a virtual PCIe expander bus tied to the appropriate NUMA node.

The message also implicitly validated that the host's NUMA topology is functioning correctly and that the GPU PCIe assignments are stable. If there had been any misconfiguration in the host's IOMMU groups or PCIe topology, the numa_node values might have been inconsistent or missing.

The Thinking Process Visible in the Conversation

The chain of reasoning that led to this message is a textbook example of systematic debugging. The assistant had identified that the VM's topology was worse than bare metal in two specific ways (missing NUMA affinity and missing NODE/SYS distinction). Rather than guessing at solutions, the assistant formulated a hypothesis: that by correctly assigning NUMA affinity in the VM, the topology could be improved. The first step in testing this hypothesis was to determine the physical NUMA mapping of the GPUs.

The assistant's thinking, visible in message 371, shows an understanding that the VM's current PHB topology is a "flattened" representation that loses the NUMA hierarchy information. The assistant recognized that Proxmox's hostpci mappings don't automatically associate GPUs with specific NUMA nodes, and that this information must be explicitly configured. The proposed solution — using pcie-expander-bus in QEMU to create separate PCIe hierarchies tied to different NUMA nodes — was technically sophisticated but acknowledged to be difficult within Proxmox's configuration framework.

The user's response (message 372) is a model of precise, actionable data provision. The user didn't just run the command; they included the full command and its output, preserving the exact shell session. This allowed the assistant to verify the command syntax and confirm that the output was complete and correct. The inclusion of the hostname kpro6 in the shell prompt also provided useful context about which machine the command was executed on.

The Aftermath and Significance

In the messages that followed, the assistant used this NUMA mapping to cross-reference the VM's GPU PCIe bus IDs (which appeared as 0000:01:00.0 through 0000:08:00.0 in the guest) with the physical host addresses. This revealed that the VM's PCIe enumeration had renumbered the GPUs sequentially, losing the original NUMA association. The assistant then explored options for configuring NUMA affinity in the Proxmox VM configuration, including the numa parameter and CPU pinning.

Ultimately, the NUMA mapping data from message 372 became a cornerstone of the investigation. While the full reconstruction of a bare-metal-like PCIe topology in the VM proved impractical within Proxmox's constraints, the knowledge that the GPUs were physically split 4+4 across NUMA nodes informed subsequent decisions about tensor parallelism placement and communication optimization strategies. The message stands as a testament to the value of precise, low-level system interrogation in diagnosing complex virtualization performance issues — and to the collaborative rhythm between human intuition and machine analysis that characterizes effective debugging at the hardware-software boundary.