The Sysfs Hack: Manually Patching NUMA Affinity for Virtualized GPUs

In the trenches of high-performance GPU inference, sometimes the most elegant solution is not a grand architectural change but a small, surgical intervention at the kernel level. Message [msg 394] captures exactly such a moment: the assistant, after an extensive investigation into PCIe topology, IOMMU groups, and Proxmox virtualization internals, discovers that the Linux kernel's numa_node sysfs attribute for PCI devices is writable and proceeds to manually assign NUMA affinity to all 8 virtualized GPUs and their parent PCIe root ports.

This single message represents a turning point in a much larger debugging effort. The broader context is a deployment of the GLM-5-NVFP4 large language model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs running inside a Proxmox virtual machine. The team had already resolved numerous challenges—flash-attn compilation issues, CUDA toolkit version mismatches, NaN crashes during model decode—and had successfully established baseline throughput. But a critical performance bottleneck remained: cross-GPU communication was suffering because the virtualized PCIe topology did not expose the underlying physical NUMA domains.

The Problem: Invisible NUMA Domains in a Virtualized Environment

To understand why message [msg 394] matters, one must first understand the NUMA topology of the host machine. The ASUS ESC8000A-E13 motherboard, powered by dual AMD EPYC processors, distributes its 8 GPUs across two physical NUMA nodes. GPUs connected to PCIe root complexes on NUMA node 0 can communicate with each other at full PCIe Gen5 x16 bandwidth (32 GT/s), while cross-NUMA communication traverses the Infinity Fabric interconnect between CPU sockets, incurring significantly higher latency and lower bandwidth. On bare metal, nvidia-smi topo -m would clearly show this distinction, labeling intra-NUMA connections as NODE and cross-NUMA connections as SYS, allowing NCCL (NVIDIA Collective Communications Library) to make optimal routing decisions.

Inside the Proxmox VM, however, this topology information was lost. All 8 GPUs appeared as PHB (PCIe Host Bridge) connections to each other, and critically, every GPU's numa_node sysfs file read -1, indicating no NUMA affinity. This meant NCCL had no way to distinguish between a GPU on the same NUMA node and one across the socket, forcing it to use suboptimal communication algorithms that treated all cross-GPU transfers as equally expensive.

The assistant had already explored several approaches to fix this. A custom NCCL topology XML file had been created and validated, but this was a software-level workaround that NCCL had to explicitly load via the NCCL_TOPO_FILE environment variable. A more fundamental fix—using QEMU's pxb-pcie (PCIe expander bus) devices with NUMA association—would have required custom QEMU arguments that conflicted with Proxmox's automatic hostpci device management. Neither approach was fully satisfactory.

The Discovery: A Writable Kernel Interface

The breakthrough came in the messages immediately preceding [msg 394]. The assistant had been probing the VM's PCI device tree, checking numa_node values for each GPU and its parent PCIe root ports. In [msg 393], a simple test was performed:

sudo sh -c 'echo 1 > /sys/bus/pci/devices/0000:01:00.0/numa_node'

The command succeeded. The numa_node attribute, which the kernel exposes as a read-write interface for PCI devices, accepted the write and immediately reflected the new value. This was a significant discovery: the kernel does not enforce that NUMA node assignments must come exclusively from the ACPI SRAT tables or the firmware; it allows privileged users to override them at runtime through sysfs.

This is not a bug in the kernel—it is a deliberate design choice. The numa_node sysfs attribute for PCI devices is writable to support hotplug scenarios and virtualization use cases where the hypervisor may need to communicate NUMA topology information to the guest. In a Proxmox VM using Q35 chipset with PCIe passthrough, the QEMU process does not automatically populate NUMA node information for passed-through devices, leaving them with the default -1 value. But the kernel provides the escape hatch: a privileged user inside the VM can manually set these values.

The Execution: Mapping Virtual Buses to Physical NUMA Nodes

Message [msg 394] executes this discovery at scale. The assistant issues a single bash command that writes NUMA node values to 16 sysfs files: 8 for the GPUs themselves and 8 for their parent PCIe root ports.

The mapping is carefully constructed based on the physical topology analysis performed in earlier messages. In [msg 384], the assistant had correlated the Proxmox resource mapping (pro6000) with the physical PCI addresses and their NUMA domains:

Assumptions and Limitations

The assistant makes several assumptions in this message. First, it assumes that writing to the numa_node sysfs attribute will actually influence NCCL's routing decisions. While NCCL does consult the kernel's NUMA information for PCI devices, it also performs its own topology discovery through NVIDIA driver queries and PCIe access control services (ACS) checks. The sysfs override may not be sufficient if NCCL's internal topology detection overrides the kernel's NUMA assignment.

Second, the assistant assumes that the mapping from virtual PCI buses to physical NUMA nodes is correct. This mapping was derived from the Proxmox resource configuration file (/etc/pve/mapping/pci.cfg), which lists the physical PCI addresses in the order they are assigned to hostpci0-7. If the Proxmox mapping order does not correspond to the physical NUMA topology as expected, the NUMA assignments would be incorrect and could potentially worsen performance.

Third, the sysfs modification is ephemeral. It persists only until the next reboot or driver reload. The assistant does not provide a mechanism to make this change permanent, such as a systemd service or a udev rule that runs at boot time. This means the fix must be reapplied after every VM restart.

Fourth, there is an implicit assumption that setting NUMA affinity on the PCIe root ports is meaningful. The root ports at 00:10.x and 00:1c.x are virtual devices emulated by QEMU, not physical hardware. Writing NUMA node values to virtual devices may have no effect on actual memory access patterns, since the QEMU process itself does not use this information for memory allocation decisions.

The Deeper Significance

Message [msg 394] is a classic example of the "sysfs hack" pattern that appears frequently in systems engineering. When a higher-level abstraction (the Proxmox VM configuration, the QEMU machine type, the ACPI firmware tables) fails to properly communicate topology information, the engineer reaches for the lowest-level interface available—in this case, the kernel's sysfs attribute for PCI devices.

The elegance of this approach lies in its immediacy. Unlike the NCCL topology XML workaround, which requires NCCL to be explicitly configured to load the file, the sysfs modification is transparent to all userspace applications. Any tool that queries the kernel's NUMA topology—whether NCCL, numactl, or a custom CUDA program—will see the corrected values. It is a system-wide fix rather than an application-specific workaround.

However, the message also reveals the limits of what can be achieved through sysfs hacks. The fundamental problem—that the VM's virtual PCIe topology does not expose the physical NUMA domains—remains unsolved. The GPUs are still connected through QEMU's virtual PCIe root ports, not through the physical PCIe switches and root complexes of the AMD EPYC platform. The sysfs modification tells the kernel what the topology should be, but it cannot change the actual data paths that packets traverse.

The Thinking Process Visible in the Message

The message reveals a clear chain of reasoning. The assistant begins by stating the discovery: "We can manually set the numa_node sysfs attribute." This is presented as a new capability, not a previously planned step. The assistant then immediately moves to execution, constructing the bash command that sets all 16 sysfs files.

The structure of the command reflects the assistant's understanding of the topology. The GPUs are grouped by NUMA node, with comments explaining the mapping. The parent root ports are included because the assistant recognizes that NUMA affinity propagates through the PCIe hierarchy—a device's NUMA node is typically inherited from its parent bus.

The verification step at the end is crucial. The assistant does not assume the writes succeeded; it explicitly reads back the values to confirm. This attention to verification is characteristic of the session's overall approach, where every configuration change is validated before proceeding.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. The numa_node sysfs attribute for PCI devices is writable in this VM environment. This is not universally true across all kernel configurations and hypervisors, but it works here.
  2. The correct mapping from virtual PCI buses to physical NUMA nodes is confirmed. The mapping derived from the Proxmox resource configuration is validated by the successful write and readback.
  3. A working command sequence exists to restore NUMA affinity after VM boot. This can be scripted and automated.
  4. The parent PCIe root ports also accept NUMA node writes. This provides a more complete topology fix than setting GPU devices alone. The message also implicitly documents that the QEMU/Q35 virtual PCIe topology does not automatically populate NUMA node information for passed-through devices, confirming a limitation of the Proxmox virtualization stack that the team must work around.

Conclusion

Message [msg 394] is a moment of tactical victory in a larger strategic battle. The assistant discovers a low-level kernel interface that allows manual correction of NUMA topology information, executes the fix with precision, and verifies the result. It does not solve the fundamental problem—the GPUs remain isolated by their physical PCIe root complexes, preventing true P2P DMA—but it removes one layer of misconfiguration that was compounding the performance issue.

For the reader following this session, the message demonstrates a key principle of systems debugging: when the abstraction layers fail you, look for the escape hatches. The kernel's sysfs interface is one of the most powerful such escape hatches, providing direct access to device attributes that higher-level tools may not properly configure. The assistant's willingness to explore this interface, rather than accepting the -1 values as immutable, is what enables the fix.