The PCIe Topology Tango: Wrestling NUMA Affinity into a Proxmox VM for 8-GPU ML Inference

Introduction

In the world of multi-GPU machine learning inference, the physical arrangement of GPUs relative to CPU sockets and memory controllers—known as the NUMA (Non-Uniform Memory Access) topology—can make or break performance. When eight NVIDIA RTX PRO 6000 Blackwell GPUs are virtualized inside a Proxmox VM, the virtual topology presented to the guest operating system often bears little resemblance to the physical reality. Message 386 of this opencode conversation captures a pivotal moment: the assistant, having diagnosed that the VM's GPUs lack NUMA affinity entirely (showing numa_node=-1), lays out three architectural approaches to rectify this, then commits to the most radical one—bypassing Proxmox's device management entirely with raw QEMU arguments. This message is a masterclass in systems-level reasoning under constraint, revealing how deep PCIe topology knowledge, QEMU internals, and practical virtualization trade-offs converge in the pursuit of shaving microseconds off inter-GPU communication.

The Message in Full

The assistant wrote:

The VM has 2 NUMA nodes (good — from numa: 1 in the VM config), but both GPUs show numa_node=-1 (no affinity). The Q35 virtual PCIe root ports don't have NUMA affinity set.

>

There are two approaches to fix this:

>

### Approach 1: Custom QEMU args with pxb-pcie (PCIe expander bus) QEMU has a pxb-pcie device that creates a PCIe expander bus with NUMA association. But this conflicts with Proxmox's automatic hostpci management.

>

### Approach 2: Bypass hostpci entirely and use raw QEMU -device vfio-pci args We'd remove all hostpci lines and use -device vfio-pci,host=... with custom PCIe topology via -device pxb-pcie,bus_nr=...,numa_node=.... This is the most flexible but requires careful QEMU arg construction.

>

### Approach 3: NCCL topology XML (software workaround) Create an XML file that tells NCCL the real topology. This doesn't change actual hardware paths but helps NCCL choose better algorithms.

>

Let me try Approach 2 since it gives us the real topology fix. I'll construct the full QEMU args. First let me verify the QEMU version and capabilities: [bash] ssh 10.1.230.175 "cat /proc/cmdline" 2>&1 BOOT_IMAGE=/vmlinuz-6.8.0-100-generic root=UUID=e4cd6da8-09cb-4378-9577-e2053726667f ro console=tty1 console=ttyS0 pci=realloc

Why This Message Was Written: The Diagnostic Culmination

This message is the logical endpoint of a long investigative chain stretching back through dozens of earlier exchanges. The overarching problem was that tensor-parallel inference on eight GPUs was bottlenecked by inter-GPU communication latency—measured at approximately 13 microseconds per small transfer. Earlier in the conversation (see [msg 369]), the assistant had concluded that true Peer-to-Peer (P2P) DMA could not be enabled in the VM because each GPU sits on its own physical PCIe root complex on the AMD EPYC platform, and VFIO mediates all inter-GPU transfers through host memory regardless of virtual topology tricks.

However, the assistant identified a second-order problem that could be fixed: the VM presented all eight GPUs with no NUMA affinity whatsoever. On bare metal, GPUs 0–3 reside on NUMA node 0 and GPUs 4–7 on NUMA node 1, with intra-socket communication labeled NODE and cross-socket communication labeled SYS by nvidia-smi topo. NCCL (NVIDIA Collective Communications Library) uses these topology hints to optimize its communication algorithms—preferring intra-socket transfers, choosing ring or tree layouts that minimize cross-socket traffic, and tuning buffer sizes accordingly. In the VM, all GPU pairs showed PHB (PCIe Host Bridge) with no NUMA distinction, robbing NCCL of this optimization signal.

The immediate trigger for message 386 was the result from [msg 385], where the assistant ran numactl --hardware and checked numa_node for two GPU devices inside the VM. The output confirmed: the VM had two NUMA nodes (from the numa: 1 setting in the Proxmox config), but both GPUs returned numa_node=-1—meaning the virtual PCIe devices had no NUMA affinity assigned. The Q35 chipset's PCIe root ports, by default, do not carry NUMA node information. The assistant recognized this as a solvable problem and wrote message 386 to enumerate the possible solutions and commit to a path forward.

How Decisions Were Made: A Three-Option Trade-off Analysis

The assistant's decision-making in this message is structured as a classic engineering trade-off analysis, presented as three discrete approaches.

Approach 1—Custom QEMU args with pxb-pcie: The assistant identifies that QEMU has a pxb-pcie (PCIe Expander Bus) device that can be created with an associated NUMA node. This would allow attaching the eight GPUs to two virtual PCIe expander buses, each pinned to a different NUMA node, thereby giving the guest OS the topology information it needs. However, the assistant immediately flags a critical conflict: Proxmox's hostpci mechanism automatically manages the creation of PCIe devices in the VM. If one manually adds pxb-pcie devices via -args, they operate in a separate layer that can clash with Proxmox's automatic bus numbering and device placement. This approach is rejected not because it's technically impossible, but because the integration complexity with Proxmox's management layer is too high.

Approach 2—Bypass hostpci entirely with raw -device vfio-pci: This is the most radical option. Instead of using Proxmox's convenient hostpci0: mapping=pro6000,pcie=1 syntax, the assistant proposes removing all eight hostpci lines from the VM config and constructing the entire PCIe topology manually using QEMU's -device arguments. This would involve creating two pxb-pcie buses with numa_node=0 and numa_node=1, then attaching four vfio-pci devices to each bus. The assistant acknowledges this is "the most flexible but requires careful QEMU arg construction." The decision to pursue this approach reflects a willingness to trade operational convenience for architectural correctness—a hallmark of systems debugging where the root cause demands fundamental restructuring rather than surface-level patches.

Approach 3—NCCL topology XML (software workaround): This is the pragmatic fallback. Rather than changing the actual hardware topology presented to the VM, one can create an XML file that NCCL reads to understand the real physical topology. This is a purely software intervention: NCCL would think it's running on a machine with proper NUMA topology and choose optimal algorithms accordingly, even though the actual data paths remain unchanged. The assistant correctly notes that "this doesn't change actual hardware paths but helps NCCL choose better algorithms." This approach is deferred, likely because it addresses the symptom (suboptimal NCCL algorithm selection) rather than the root cause (missing NUMA affinity), and because the assistant believes Approach 2 can deliver a genuine fix.

The decision to "try Approach 2" is stated with confidence, but the assistant immediately pivots to a verification step: checking the QEMU version and capabilities by reading /proc/cmdline from the VM. This reveals a pragmatic caution—before committing to a complex QEMU arg construction, the assistant wants to confirm that the QEMU version in use supports pxb-pcie and the necessary features. The pci=realloc flag visible in the kernel command line is a remnant of an earlier BAR allocation fix (from <msg id=...> in chunk 0), confirming that the VM's kernel is already configured to handle the PCIe address space demands of eight GPUs.

Assumptions Embedded in the Message

Several assumptions underpin the reasoning in this message, some explicit and some implicit.

Assumption 1: NUMA affinity for virtual PCIe devices will improve NCCL performance. This is the core hypothesis. The assistant assumes that if the VM exposes proper NUMA topology (GPU0-3 on NUMA 0, GPU4-7 on NUMA 1), NCCL will select better communication algorithms and achieve lower latency. This is reasonable based on NCCL's documented behavior, but it's worth noting that the actual data path still goes through VFIO-mediated host memory—the physical latency floor of ~13µs may not change. What would change is NCCL's algorithm selection: it might choose different ring orderings, buffer sizes, or parallelism strategies that reduce the number of transfers or better overlap communication with computation. The assistant implicitly assumes this optimization signal is valuable enough to justify the engineering effort of Approach 2.

Assumption 2: The QEMU version supports pxb-pcie and numa_node assignment. The assistant hasn't verified this yet—that's why the first action after the message is to check /proc/cmdline and presumably the QEMU version. On older QEMU builds (pre-4.0 or certain distribution backports), pxb-pcie may not be available or may not support NUMA node association. The assistant is proceeding optimistically but building in a verification step.

Assumption 3: The mapping order in pci.cfg correctly groups NUMA-local GPUs. In [msg 384], the assistant decoded the pro6000 mapping and found that hostpci0-3 (VM GPUs 0-3) map to physical GPUs on NUMA 1, while hostpci4-7 (VM GPUs 4-7) map to physical GPUs on NUMA 0. The assistant assumes this grouping is correct and stable—that the four GPUs on each NUMA node are indeed the ones that should share a virtual PCIe expander bus. This is a reasonable inference from the host-side data, but it depends on the physical PCIe slot layout and the BIOS's NUMA node assignment, which could theoretically change after a hardware reconfiguration.

Assumption 4: The NCCL topology XML workaround is a viable fallback. The assistant presents Approach 3 as a simpler alternative, but it carries its own assumptions: that the application uses NCCL (which sglang does for tensor-parallel all-reduce), that NCCL will read and respect the XML file, and that the XML format hasn't changed between NCCL versions. These are reasonable but unverified.

Input Knowledge Required to Understand This Message

To fully grasp the reasoning in message 386, a reader needs substantial background knowledge spanning multiple domains:

PCIe topology and NUMA architecture: Understanding that PCIe devices are attached to PCIe root complexes, which are integrated into CPU sockets (NUMA nodes). Devices on the same socket communicate via the PCIe fabric within that socket (labeled NODE by nvidia-smi topo), while devices on different sockets must traverse the inter-socket interconnect (labeled SYS). The distinction matters because SYS paths have higher latency and lower bandwidth.

QEMU/KVM virtualization internals: Knowledge of QEMU's device model, particularly the -device syntax, vfio-pci for PCIe passthrough, pxb-pcie for creating virtual PCIe expander buses, and how numa_node is assigned to virtual devices. Also understanding that Proxmox's hostpci syntax is a convenience layer that generates QEMU -device arguments automatically, and that custom -args can supplement or override these.

NCCL topology discovery: NCCL (NVIDIA Collective Communications Library) probes the system topology at initialization time, reading nvidia-smi topo output and system NUMA information to build an internal topology graph. It uses this graph to select communication algorithms (ring, tree, all-to-all) and to assign GPU ranks to communication channels. An NCCL topology XML file can override the auto-detected topology.

The GLM-5-NVFP4 deployment context: The broader goal is to run the GLM-5-NVFP4 model with tensor parallelism across eight GPUs using sglang. Tensor parallelism requires frequent all-reduce operations between GPUs, making inter-GPU latency a critical performance factor. The 13µs latency floor identified earlier is the product of VFIO-mediated staging through host memory.

Proxmox VE configuration syntax: Understanding the hostpci mapping mechanism, the pci.cfg file format, and how qm set and VM config files work. The mapping=pro6000 syntax references a resource mapping that defines which physical PCI devices to passthrough and in what order.

Output Knowledge Created by This Message

Message 386 produces several forms of output knowledge:

A decision framework for NUMA topology repair: The three-approach taxonomy (pxb-pcie with hostpci, raw vfio-pci with pxb-pcie, NCCL XML workaround) serves as a reusable decision tree for anyone facing similar NUMA affinity issues in KVM/Proxmox GPU VMs. Each approach is characterized by its complexity, its integration risk with Proxmox, and its fidelity to the physical topology.

A concrete action plan: The assistant commits to Approach 2 and begins executing it by checking QEMU capabilities. This creates a clear next step that the user (and the assistant in subsequent messages) can follow.

A verification of the VM's current state: The /proc/cmdline output confirms that pci=realloc is active, which was a critical fix from earlier in the conversation (the BAR allocation failure when migrating to Q35 chipset). This confirms that the VM's PCIe configuration is stable enough to proceed with further topology manipulation.

An implicit documentation of QEMU's NUMA limitations: The observation that "Q35 virtual PCIe root ports don't have NUMA affinity set" is a useful piece of systems knowledge. It explains why simply using hostpci with pcie=1 on Q35 is insufficient for NUMA-aware GPU topology—the virtual root ports themselves lack NUMA association, and this cannot be fixed through the Proxmox config syntax alone.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message reveals a structured, multi-layered thought process:

Layer 1—Diagnostic synthesis: The assistant synthesizes two pieces of information: (a) the VM has two NUMA nodes (from the numa: 1 config), and (b) both GPUs show numa_node=-1. The contradiction between these two facts is the crux—the VM knows about NUMA but hasn't assigned devices to NUMA nodes.

Layer 2—Root cause attribution: The assistant correctly identifies the root cause: "The Q35 virtual PCIe root ports don't have NUMA affinity set." This is not a bug or a misconfiguration—it's a default behavior of QEMU's Q35 machine type. The virtual PCIe root ports that Proxmox creates for hostpci devices are not associated with any NUMA node unless explicitly configured.

Layer 3—Solution space enumeration: Rather than jumping to a single solution, the assistant enumerates the full solution space. This is characteristic of strong systems thinking: generate all viable options, evaluate their trade-offs, then select. The three approaches span the spectrum from "modify existing mechanism" (Approach 1) through "replace existing mechanism" (Approach 2) to "work around the symptom" (Approach 3).

Layer 4—Trade-off evaluation: Each approach is evaluated against a consistent set of criteria: feasibility (does it conflict with Proxmox?), completeness (does it fix the root cause or just the symptom?), and risk (how much can go wrong?). Approach 1 is rejected for integration conflict. Approach 3 is deferred as a fallback. Approach 2 is selected as the primary path.

Layer 5—Execution with verification: Having selected Approach 2, the assistant doesn't immediately dive into constructing QEMU args. Instead, it takes a verification step: checking the QEMU version and capabilities. This reveals an awareness that the plan depends on unverified prerequisites. The /proc/cmdline check is a lightweight way to confirm the VM's kernel is in a known good state before making invasive changes.

A subtle meta-cognitive thread: The assistant's statement "Let me try Approach 2" is notable for its tentative framing. Despite presenting the decision confidently, the "try" acknowledges uncertainty—this is an experimental intervention, not a guaranteed fix. The assistant is managing expectations, both for the user and for itself.

Potential Mistakes and Incorrect Assumptions

While the reasoning is sound, several points warrant critical examination:

The NCCL topology XML workaround may be undervalued. The assistant presents Approach 3 as a "software workaround" that doesn't change actual hardware paths. However, for NCCL's purposes, the perceived topology is what matters—NCCL makes algorithm decisions based on topology information, not on raw latency measurements. If the XML file correctly describes the NUMA topology, NCCL would choose the same algorithms it would on bare metal. The actual data paths would still go through VFIO staging, but NCCL's algorithm choices would be optimal for that topology. In practice, this might yield most of the benefit of Approach 2 with far less complexity. The assistant may be overvaluing "real" topology fixes over pragmatic software workarounds.

The assumption that pxb-pcie NUMA assignment will propagate to nvidia-smi topo. Even if the assistant successfully creates two pxb-pcie buses with NUMA node assignments and attaches four GPUs to each, it's not guaranteed that nvidia-smi topo inside the VM will report NODE and SYS labels matching the bare metal topology. The NVIDIA driver's topology detection logic may not fully respect QEMU's virtual PCIe hierarchy—it might still report PHB or PIX depending on how it interprets the virtual bus structure. The assistant is assuming that virtual PCIe expander buses will be treated analogously to physical PCIe switches, which may not hold.

The ordering assumption in the mapping file. The assistant assumes that hostpci0 gets the first entry in the pro6000 mapping, hostpci1 gets the second, and so on. While this is how Proxmox resource mappings work, the assistant didn't empirically verify the mapping by, for example, checking which physical GPU's UUID corresponds to which VM GPU index. The earlier investigation in [msg 384] relied on the order of entries in pci.cfg, which is correct but should have been cross-validated against GPU UUIDs.

The feasibility of mixing custom -device args with Proxmox management. Approach 2 involves removing all hostpci lines and using -device vfio-pci directly. While technically possible, this creates a maintenance burden: any changes to the GPU configuration (adding/removing GPUs, changing passthrough options) would require manually editing the QEMU args rather than using Proxmox's management UI or API. The assistant doesn't discuss this operational cost.

Conclusion

Message 386 captures a moment of architectural decision-making in a complex systems debugging journey. The assistant, having traced the performance bottleneck to missing NUMA affinity in the virtual PCIe topology, evaluates three approaches with clear reasoning about trade-offs and commits to the most ambitious one. The message reveals deep knowledge of QEMU internals, PCIe topology, NCCL behavior, and Proxmox configuration—and more importantly, it demonstrates how to synthesize these domains into a coherent action plan. Whether Approach 2 ultimately succeeds or fails, the analytical framework established in this message—diagnose the gap, enumerate the options, evaluate trade-offs, verify prerequisites—is a model for systematic debugging in virtualized GPU environments. The 13-microsecond latency floor may not be eliminated by this intervention, but the pursuit of proper NUMA topology is a necessary step in understanding and optimizing the full communication path.