The Sysfs Gambit: A Desperate Attempt to Fix GPU NUMA Topology from Inside the VM

Introduction

In the long and arduous journey to optimize cross-GPU communication for an 8-GPU inference deployment, there comes a moment when conventional solutions have failed, architectural approaches have been blocked by hardware constraints, and the engineer is left grasping for anything that might work. Message [msg 393] captures exactly such a moment: a single, almost defiant bash command that attempts to brute-force a NUMA topology fix by writing directly to a kernel sysfs file.

The message is deceptively simple:

[assistant] [bash] ssh 10.1.230.175 "sudo sh -c 'echo 1 > /sys/bus/pci/devices/0000:01:00.0/numa_node' 2>&1 && cat /sys/bus/pci/devices/0000:01:00.0/numa_node"
1

A single command, a single digit of output. But behind this terse exchange lies a rich story of escalating constraints, architectural dead ends, and the kind of pragmatic hacking that emerges when every proper solution has been exhausted.

The Motivation: Why This Message Was Written

To understand why the assistant resorted to writing directly to a sysfs file, we must trace the chain of reasoning that led to this point. The overarching problem was severe cross-GPU communication latency in a Proxmox virtual machine hosting 8 NVIDIA RTX PRO 6000 Blackwell GPUs. Benchmarking in [msg 374] through [msg 392] had revealed that NCCL's P2P (Peer-to-Peer) transfers were falling back through system memory rather than using direct GPU-to-GPU DMA, crippling the throughput of tensor-parallel inference.

The root cause was topological: in the bare-metal host, the 8 GPUs were split across two physical NUMA domains (GPUs 0–3 on NUMA 1, GPUs 4–7 on NUMA 0), with each GPU sitting on its own dedicated PCIe root complex. This motherboard design—standard for the ASUS ESC8000A-E13—maximizes per-GPU bandwidth but creates an insurmountable barrier for P2P DMA inside a virtual machine. As the assistant had discovered in earlier chunks, the VFIO/IOMMU subsystem cannot grant direct P2P access across separate IOMMU groups, and each GPU belonged to its own group because they shared no common PCIe switch.

The assistant had already pursued multiple avenues. It had migrated the VM from the legacy i440FX chipset to Q35 with proper PCIe passthrough (pcie=1), fixed BAR allocation failures with pci=realloc, attempted ACS (Access Control Services) disable on the host to merge IOMMU groups, and even explored dangerously insecure kernel parameters like vfio_iommu_type1.allow_unsafe_interrupts. None of these worked. The hardware topology was immutable: each GPU on its own root complex meant P2P DMA across all 8 GPUs was fundamentally impossible in a VM.

Faced with this architectural dead end, the assistant pivoted to a software-based workaround: the NCCL topology XML file. In [msg 390], it crafted a custom XML file that told NCCL the real NUMA topology—GPUs 0–3 on NUMA node 1, GPUs 4–7 on NUMA node 0—regardless of what the virtual hardware reported. This was tested successfully in [msg 391] with NCCL accepting the override. But the assistant was not satisfied with a purely software-level fix. In [msg 392], it began exploring whether NUMA affinity could be set at the VM level, asking: "can we also set NUMA affinity on the GPU PCI devices from inside the VM?"

The answer came back starkly: all 8 GPUs showed numa_node=-1. The virtual PCIe root ports created by QEMU had no NUMA affinity assigned. This is where message [msg 393] enters the story.

The Command: What It Actually Does

The command is a study in minimalism. It SSHes into the VM at 10.1.230.175 and executes a shell pipeline that:

  1. Uses sudo sh -c to gain root privileges and execute a command string
  2. Writes the value 1 to the file /sys/bus/pci/devices/0000:01:00.0/numa_node
  3. Redirects stderr to stdout (2>&1) to capture any error messages
  4. On success (&&), reads back the file to confirm the write took effect The target device 0000:01:00.0 is GPU 0 in the VM's PCI topology. In the Q35-based virtual machine, the first four GPUs (hostpci0–3) appear on virtual PCI buses 01–04, attached to the Q35's PCIe root port controller at 00:10.x. The numa_node sysfs attribute is a kernel interface that reports (and in some configurations, allows setting) the NUMA node affinity for a PCI device. A value of -1 means "no affinity" or "unknown"; a value of 0 or 1 assigns the device to a specific NUMA domain. The output is simply 1, confirming that the write succeeded and the file now contains the value 1. At first glance, this appears to have worked: the kernel accepted the write and the NUMA node for GPU 0 is now set to 1.

Assumptions Embedded in the Approach

This command rests on several assumptions, some of which are questionable:

First, it assumes that writing to the numa_node sysfs file actually changes the kernel's NUMA affinity for the device, rather than being a read-only attribute that silently ignores writes. On many kernel configurations, numa_node is a read-only attribute derived from ACPI tables or PCIe topology; writing to it may appear to succeed (the write returns without error) but have no effect on actual memory allocation or DMA routing.

Second, it assumes that setting the NUMA node on a single GPU is meaningful in isolation. The command only targets GPU 0 (0000:01:00.0). Even if the write has real effect, the other 7 GPUs remain at numa_node=-1. The assistant would need to repeat this for all 8 GPUs, and the NUMA assignments would need to match the real physical topology (GPUs 0–3 on node 1, GPUs 4–7 on node 0) for NCCL to benefit.

Third, it assumes that the sysfs interface is the correct layer at which to fix NUMA topology. In reality, NUMA affinity for PCI devices is established much earlier in the boot process, during ACPI table parsing and PCI enumeration. A late-stage sysfs write is at best a hint and at worst completely ignored by the subsystems that matter—the IOMMU, the memory allocator, and the interrupt controller.

Fourth, and most subtly, the command assumes that the problem is merely informational—that if NCCL and the GPU drivers believe the GPUs have NUMA affinity, they will behave as if the affinity is real. This is the same assumption underlying the NCCL topology XML approach, but applied at the kernel level. The NCCL topo file is a well-documented override mechanism; sysfs manipulation is not.

Input Knowledge Required

To understand this message, a reader needs substantial background knowledge spanning several domains:

Output Knowledge Created

The message produces a single, ambiguous piece of knowledge: the sysfs numa_node file for GPU 0 now contains 1. But what does this actually mean?

On the positive side, it confirms that the sysfs attribute is writable in this kernel configuration. Not all kernels allow writes to numa_node; some have strict read-only enforcement. The fact that the write succeeded without a "Permission denied" or "Invalid argument" error is itself useful information.

However, the message does not confirm that the NUMA affinity has actually changed. The kernel may accept the write but ignore the value during actual device operations. The numa_node file might be a one-way trap: you can write to it, but the value is never consulted by the memory allocator or the IOMMU. The only way to verify real effect would be to observe changes in memory allocation patterns, interrupt routing, or NCCL behavior—none of which are tested in this message.

The message also creates implicit knowledge about the assistant's problem-solving strategy: when architectural solutions fail, it is willing to try low-level, potentially unreliable hacks. This is a pragmatic engineering mindset, but one that carries risks of silent failure.

The Thinking Process Visible in the Reasoning

Although the message itself contains no explicit reasoning (it is a single command with no commentary), the thinking process is visible in the surrounding context. In [msg 392], the assistant had just finished testing the NCCL topology XML approach and was explicitly asking: "can we also set NUMA affinity on the GPU PCI devices from inside the VM?" This question reveals a specific line of thought: if the NCCL topo file works at the application layer, perhaps a similar override at the kernel layer would be even more effective.

The assistant then ran a diagnostic loop over all 8 GPUs, confirming that every one showed numa_node=-1, and also checked the root port device (0000:00:10.0), which also showed -1. This systematic enumeration suggests the assistant was looking for any writable NUMA attribute in the PCI hierarchy.

The jump from "check what the values are" to "try writing a new value" is a natural one for an experienced Linux engineer. When a system configuration file shows a default or incorrect value, the instinct is to change it. The assistant appears to be testing a hypothesis: "What if the NUMA node is just a sysfs attribute that we can override?" The success of the write (no error) is encouraging, but the assistant does not yet know if the override is effective.

Mistakes and Incorrect Assumptions

The most significant potential mistake is the assumption that writing to numa_node has real effect. In many Linux kernel configurations, numa_node is a read-only attribute derived from the ACPI SLIT (System Locality Information Table) or from the PCIe hierarchy as reported by the firmware. The sysfs file may be implemented as a dev_attr that simply returns a stored value without a corresponding setter. Even if a setter exists, changing the NUMA node after device initialization may not update the IOMMU page tables, DMA masks, or memory allocation policies that actually determine where a device's memory is allocated.

A second issue is the lack of verification. The command checks that the write succeeded by reading back the file, but this only confirms that the value was stored in the sysfs attribute—not that any kernel subsystem is acting on it. A proper test would require running a memory-intensive GPU workload and checking which NUMA node's memory is being used, or examining /proc/ or numactl output for the device's memory policy.

A third subtlety: even if the sysfs override works, setting GPU 0 to NUMA node 1 while leaving GPUs 1–7 at -1 creates an inconsistent state. The assistant would need to set all 8 GPUs to their correct NUMA nodes, and the mapping would need to match the NCCL topology XML (GPUs 0–3 on node 1, GPUs 4–7 on node 0). A partial fix could cause worse behavior than no fix at all.

Conclusion

Message [msg 393] is a small but revealing moment in a much larger debugging saga. It captures the transition from architectural problem-solving to pragmatic hacking—the point at which an engineer, having exhausted proper solutions, begins probing the system's low-level interfaces for any lever that might move the needle. The command is technically successful (it writes a value without error), but the real question—whether the override has any effect on GPU memory allocation or NCCL communication performance—remains unanswered within this message.

The message also illustrates a broader truth about systems engineering: the boundary between configuration and hack is often just a matter of whether the interface was designed for the purpose. The NCCL topology XML file is a documented, supported override mechanism. The sysfs numa_node write is an undocumented, unsupported one. Both aim to achieve the same goal, but only the former carries any guarantee of effectiveness.

In the end, this single line of bash—echo 1 > /sys/bus/pci/devices/0000:01:00.0/numa_node—represents the assistant's willingness to descend from the clean abstractions of application-layer configuration into the messy, ambiguous world of kernel internals. It is a testament to the depth of the P2P DMA problem that such measures were even considered.