The Moment of Verification: When Manual NUMA Node Assignments Failed to Move the Needle

In the long and arduous journey to optimize GPU-to-GPU communication across eight NVIDIA RTX PRO 6000 Blackwell GPUs running inside a Proxmox virtual machine, message [msg 395] stands as a quiet but pivotal moment of reckoning. It is a single, almost laconic exchange: the assistant runs nvidia-smi topo -m and receives a stark table of PHB entries. But beneath this brief command-and-output lies a dense layer of technical reasoning, a hypothesis being tested, and a critical falsification that reshapes the entire trajectory of the debugging effort.

The Context: A Battle Against Virtualization Overhead

To understand why this message matters, one must first appreciate the broader struggle. The user is running a demanding machine learning inference workload—deploying the GLM-5-NVFP4 model using SGLang with tensor parallelism across eight GPUs—inside a Proxmox virtualized environment. The host machine is an ASUS ESC8000A-E13 server with dual AMD EPYC processors, where each of the eight RTX PRO 6000 GPUs sits on its own dedicated PCIe root complex. This physical topology, while maximizing per-GPU bandwidth, creates a fundamental problem for peer-to-peer (P2P) DMA transfers: each GPU is isolated in its own IOMMU group, and the VFIO driver cannot grant direct cross-GPU memory access.

The practical consequence is severe. Without P2P, every GPU-to-GPU communication must traverse the host's PCIe root complex and system memory rather than happening directly between GPU BARs. This adds latency, consumes CPU and memory bandwidth, and dramatically reduces the throughput of collective operations like all-reduce that are the lifeblood of tensor-parallel inference. The assistant had spent the preceding messages investigating and attempting various remedies: modifying the Proxmox host kernel to enable IOMMU passthrough, migrating the VM from the legacy i440FX chipset to Q35 with proper PCIe device passthrough, fixing BAR allocation failures, attempting to disable Access Control Services (ACS) to merge IOMMU groups, and ultimately discovering that the hardware topology—each GPU on its own root complex—fundamentally prevents P2P regardless of software configuration.

The Hypothesis: Can Sysfs Override NVIDIA's Topology View?

By message [msg 393], the assistant had discovered something intriguing: the Linux kernel's sysfs interface allows writing to the numa_node attribute of PCI devices. A quick test showed that echo 1 > /sys/bus/pci/devices/0000:01:00.0/numa_node succeeded and the value was reflected on readback. This raised a tantalizing possibility: if the VM's Q35 chipset placed the eight GPUs across two virtual PCIe root port groups (buses 01-04 on 00:10.x and buses 05-08 on 00:1c.x), and if the kernel allowed manual assignment of NUMA affinity, perhaps one could trick the NVIDIA driver and NCCL into seeing a topology that matched the physical reality.

In message [msg 394], the assistant acted on this hypothesis, executing a comprehensive set of sysfs writes:

The Message: A Verification That Becomes a Falsification

Message [msg 395] is that test. The assistant runs:

nvidia-smi topo -m

The output is a 9×9 matrix showing the connectivity topology between all eight GPUs, plus columns for CPU affinity, NUMA affinity, and GPU NUMA ID. Every off-diagonal cell reads PHB—PCIe Host Bridge. Every GPU's CPU affinity spans all 120 CPUs (0-119). Every GPU's NUMA affinity shows both nodes (0-1). And every GPU's NUMA ID column reads N/A.

This is the moment of truth, and it is devastating. The manual numa_node writes had zero effect on how nvidia-smi topo -m reports the topology. The NVIDIA driver, it turns out, does not consult the kernel's numa_node sysfs attribute when constructing its topology matrix. Instead, it derives the topology from the actual PCIe hierarchy as seen through the NVIDIA kernel driver's own device interrogation—the bus numbers, the bridge relationships, the ACPI tables, and the physical slot information exposed by the GPU's VBIOS and PCIe configuration space.

Why It Failed: The Architecture of nvidia-smi topo -m

Understanding why the sysfs approach failed requires insight into how nvidia-smi topo -m works. The tool queries the NVIDIA kernel driver (nvidia.ko or nvidia_uvm.ko), which maintains its own internal topology map built during driver initialization. This map is constructed by walking the PCIe hierarchy: examining each GPU's parent bridge, grandparent bridge, and so on up to the root complex. The driver classifies relationships into several categories:

The Assumptions Under Test

The assistant made a reasonable but ultimately incorrect assumption: that nvidia-smi topo -m derives its topology information from the kernel's PCI device NUMA affinity. This assumption was grounded in the observation that the numa_node attribute exists precisely to tell the system which NUMA node a PCI device is connected to. If the kernel can be told that GPU0 is on NUMA node 1 and GPU4 is on NUMA node 0, surely the NVIDIA driver would pick up on this and report a differentiated topology?

The assumption failed because the NVIDIA driver does not use the kernel's numa_node for topology reporting. The driver performs its own PCIe bus walk during initialization, independent of the kernel's NUMA affinity framework. This is a design choice: the driver needs to know the actual hardware topology for P2P enablement decisions, which depend on PCIe switch presence and ACS capabilities, not on software-assigned NUMA hints.

Input Knowledge Required

To fully grasp this message, the reader needs several layers of context:

  1. The PCIe topology problem: Understanding that PHB means all GPUs are on separate root complexes with no shared switch, making P2P impossible.
  2. The sysfs experiment: Knowing that in [msg 394], the assistant manually wrote NUMA node values to each GPU's sysfs entry.
  3. The nvidia-smi topo -m output format: Recognizing that the matrix shows connectivity types, and that PHB is the worst possible classification for P2P performance.
  4. The NUMA affinity columns: Understanding that "0-119" CPU affinity and "0-1" NUMA affinity for all GPUs means the VM sees all GPUs as equally distant from all CPUs—a flat topology with no NUMA differentiation.
  5. The N/A GPU NUMA ID: Recognizing that the NVIDIA driver itself reports no NUMA association for these devices, confirming that the sysfs writes did not propagate into the driver's topology model.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. Sysfs numa_node writes do not affect nvidia-smi topo -m. The NVIDIA driver builds its topology independently of the kernel's NUMA affinity hints.
  2. The VM's virtual PCIe topology is flat. All eight GPUs appear as PHB to each other, confirming that the Q35 emulation presents each GPU on its own virtual root port with no shared switch.
  3. NUMA affinity is not being propagated. Despite the VM having two NUMA nodes (from numa: 1 in the Proxmox config) and despite the manual sysfs writes, the NVIDIA driver reports no NUMA affinity for any GPU.
  4. The NCCL topology XML approach remains the only viable software workaround. Since nvidia-smi topo cannot be influenced by sysfs, the earlier approach of using NCCL_TOPO_FILE to supply a custom topology XML to NCCL (tested in [msg 391]) becomes the primary path forward.

The Broader Significance

This message is a classic example of the scientific method in systems debugging. A hypothesis is formed (manual NUMA assignment will fix topology reporting), a prediction is made (nvidia-smi topo -m will show differentiated NUMA affinity), an experiment is designed and executed (the sysfs writes followed by the topology query), and the hypothesis is falsified (the output shows no change). The assistant does not overreact or despair—the message is delivered matter-of-factly, without commentary. But the implication is clear: the path of manipulating kernel NUMA attributes is a dead end.

The message also illustrates a deeper truth about virtualized GPU environments: the hypervisor's PCIe emulation creates a topology that is a simplified abstraction of the physical hardware. In this abstraction, all GPUs are equally distant from each other and from all CPUs. The physical topology—with its NUMA domains, its PCIe switches, its Gen5 x16 links—is invisible to the guest. The NVIDIA driver, which was designed for bare-metal environments where the PCIe topology is a faithful representation of the hardware, has no mechanism to accept topology corrections from the guest kernel.

The Path Forward

After this message, the assistant pivots decisively. The NCCL topology XML file created earlier becomes the primary workaround. This file, placed at ~/nccl_topo.xml, tells NCCL that GPUs 0-3 are on NUMA node 1 and GPUs 4-7 are on NUMA node 0, with all links at PCIe Gen5 x16 speed. NCCL, unlike nvidia-smi topo -m, accepts this override via the NCCL_TOPO_FILE environment variable and will use it for its collective communication algorithm selection. While this does not enable actual P2P DMA—the hardware topology still prevents that—it at least allows NCCL to choose optimal tree and ring algorithms based on the correct NUMA boundaries, potentially improving all-reduce performance by keeping traffic within NUMA domains where possible.

The assistant also continues to explore deeper workarounds, including investigating vfio_iommu_type1.allow_unsafe_interrupts and nv_peer_mem as potential (if insecure) mechanisms to force cross-GPU P2P in the KVM environment. But the fundamental constraint remains: on the ASUS ESC8000A-E13 motherboard, each GPU is physically attached to its own PCIe root complex, and no amount of software configuration can create a shared PCIe switch where none exists in hardware.

Conclusion

Message [msg 395] is a quiet but crucial inflection point in a complex debugging session. In just a few lines, it delivers the experimental result that closes one line of inquiry and redirects effort toward more promising approaches. It exemplifies the iterative, hypothesis-driven nature of systems debugging—where progress comes not only from successful fixes but also from clear, unambiguous falsifications that prevent wasted effort on dead-end paths. The PHB matrix tells a story of hardware constraints that software alone cannot overcome, and it sets the stage for the creative workarounds that follow.