The Sysfs Mirage: When Software Topology Collides with Hardware Reality in Virtualized GPU Clusters

In the relentless pursuit of closing the performance gap between bare-metal and virtualized GPU inference, the assistant's message at index 396 represents a pivotal moment of reckoning. After an extensive journey through Proxmox host kernel modifications, VM chipset migrations, BAR allocation fixes, and ACS disable attempts—all aimed at enabling Peer-to-Peer (P2P) DMA between eight RTX PRO 6000 Blackwell GPUs—the assistant arrives at a sobering conclusion: the virtualization layer fundamentally obscures the physical PCIe topology, and no amount of sysfs manipulation can fully restore it.

The Message in Full

The assistant's message begins with a diagnostic report:

Still all PHB and NUMA Affinity: 0-1. The nvidia-smi topo uses its own detection logic that looks at the PCIe topology structure (which root port, which bridge), not just the sysfs numa_node. Since all GPUs are behind root ports on the same Q35 root complex (bus 0x00), nvidia-smi sees them all as PHB peers regardless of the sysfs numa_node setting.

The assistant then proposes a two-pronged strategy:

The sysfs numa_node fix + NCCL topo XML together is our best bet. The sysfs helps CUDA/PyTorch allocate memory on the right NUMA node, and the NCCL topo XML tells NCCL the real topology for communication planning. Even though nvidia-smi doesn't show it, NCCL will use the topo XML.

Finally, the assistant makes the sysfs changes persistent by creating an /etc/rc.local script that will re-apply the NUMA affinity settings on every boot, and executes the command to write this file to the remote VM.

The Context: A Long Road to Nowhere Fast

To understand why this message matters, we must trace the path that led here. The broader session (Segment 3) had been investigating and attempting to enable P2P DMA for GPUs in a Proxmox VM. The user had deployed GLM-5-NVFP4—a massive Mixture-of-Experts model—across 8 RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism. Early benchmarks revealed abysmal cross-GPU communication performance, with nvidia-smi topo -m showing all GPU-to-GPU links as PHB (PCIe Host Bridge)—the worst possible topology classification, indicating that every pair of GPUs must communicate through the host CPU rather than via direct P2P DMA.

The team had already attempted numerous fixes: modifying the Proxmox host kernel command line to enable IOMMU passthrough (amd_iommu=on iommu=pt), migrating the VM from the legacy i440FX chipset to Q35 with proper PCIe device passthrough, fixing BAR allocation failures with pci=realloc, and even attempting to disable ACS (Access Control Services) in the BIOS to merge IOMMU groups. Each effort peeled back another layer of abstraction, only to reveal a deeper hardware constraint.

The critical discovery was that the ASUS ESC8000A-E13 motherboard places each GPU on its own dedicated PCIe root complex. This design maximizes per-GPU bandwidth (each gets a full Gen5 x16 link) but creates separate IOMMU domains that fundamentally prevent VFIO from enabling direct P2P DMA across GPUs in a virtualized environment. No amount of kernel parameter tweaking or BIOS modification could merge what the hardware had physically separated.

Why This Message Was Written

This message was written as a direct response to the failure of the sysfs numa_node approach. In the preceding message ([msg 395]), the assistant had eagerly checked whether nvidia-smi topo -m would reflect the newly applied sysfs NUMA node assignments. The answer was a definitive no. The assistant needed to:

  1. Acknowledge the limitation of the sysfs approach transparently, rather than pretending the fix had worked.
  2. Explain the mechanism behind the failure—that nvidia-smi topo uses its own topology detection logic based on the PCIe tree structure, not the sysfs numa_node attribute.
  3. Salvage the approach by re-framing what the sysfs fix does accomplish versus what it doesn't.
  4. Integrate the two workarounds (sysfs + NCCL topo XML) into a coherent combined strategy.
  5. Make the fix persistent by creating an rc.local script, since sysfs attributes reset on reboot. The reasoning is driven by a pragmatic engineering mindset: when the ideal solution (true P2P DMA via IOMMU groups) is blocked by hardware topology, the next best thing is to approximate the topology information at multiple software layers so that CUDA memory allocation and NCCL communication planning can make better decisions, even if the hardware layer doesn't support direct peer-to-peer transfers.

How Decisions Were Made

The decision-making in this message reflects a sophisticated understanding of the GPU software stack's layered architecture. The assistant recognized that different components use different sources of topology information:

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

Explicit assumption: "The sysfs helps CUDA/PyTorch allocate memory on the right NUMA node." This is a reasonable assumption—CUDA's memory allocation routines do consult the sysfs numa_node attribute when deciding where to allocate host-pinned memory. However, the extent to which this matters depends on whether the application actually uses host-pinned memory (e.g., for data transfers) versus device-local memory. For a pure inference workload where model weights live entirely on GPU VRAM, the NUMA affinity of host-pinned memory may have limited impact.

Explicit assumption: "NCCL will use the topo XML." This is correct—NCCL's NCCL_TOPO_FILE environment variable is a well-documented feature that overrides the detected topology. However, NCCL's internal algorithms also have fallback paths; if the XML describes a topology that NCCL considers invalid or inconsistent with the detected PCIe hierarchy, it may fall back to its own detection.

Implicit assumption: The Q35 chipset's two PCIe root port groups (00:10.x and 00:1c.x) correspond to the two physical NUMA domains. This was verified earlier in the session by correlating the Proxmox mapping order with the physical PCIe addresses. The assistant had mapped hostpci0-3 (VM buses 01-04) to physical GPUs on NUMA 1, and hostpci4-7 (VM buses 05-08) to physical GPUs on NUMA 0. This mapping is correct for this specific hardware configuration but would need to be re-verified if the GPU order in the Proxmox mapping changes.

Implicit assumption: The rc.local script will execute successfully at boot. This depends on the script being executable, having the correct shebang, and the system having rc.local support enabled. The assistant created the file with #!/bin/bash and made it executable with chmod +x, which are the standard requirements.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption is revealed in the message itself: that setting the sysfs numa_node attribute would cause nvidia-smi topo -m to show the correct topology. The assistant's earlier enthusiasm in [msg 394] ("We can manually set the numa_node sysfs attribute") was tempered by the reality check in [msg 395] and this message. The nvidia-smi topo output stubbornly showed all PHB with NUMA Affinity: 0-1—the same as before the sysfs changes.

This is a subtle but important mistake. The sysfs numa_node attribute is a consumer of topology information, not a source of it. It reflects what the kernel's PCI subsystem has determined about the device's NUMA affinity based on the ACPI SRAT (System Resource Affinity Table) and the PCIe hierarchy. Writing to it changes the attribute value, but the NVIDIA driver's topology detection doesn't read this attribute—it walks the PCIe tree structure independently. The assistant correctly identified this mechanism in the message ("nvidia-smi topo uses its own detection logic that looks at the PCIe topology structure"), but this realization came after the sysfs manipulation had already been performed.

Another potential issue: the rc.local script writes to sysfs files that may not exist at boot time if the PCI devices haven't been enumerated yet. The script uses 2>/dev/null to suppress errors, which is good practice, but it means failures are silently ignored. A more robust approach would be to use a systemd service with After=sys-devices-... dependencies, or a udev rule that triggers when the specific PCI devices appear. However, for a development VM where silent failure is acceptable, the rc.local approach is reasonable.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. The PCIe topology hierarchy: How root complexes, root ports, bridges, and endpoints relate. The Q35 chipset's PCIe root port layout (00:10.x for the first group, 00:1c.x for the second) and how these map to virtual PCI buses in the VM.
  2. NUMA architecture: What NUMA nodes are, how memory affinity works, and why cross-NUMA memory access is slower than local access. The AMD EPYC platform's NUMA topology (two NUMA domains, each with 60 CPU threads and ~216 GB of RAM).
  3. The NVIDIA driver's topology classification: The difference between PHB (PCIe Host Bridge), PXB (PCIe Switch), NODE (same NUMA), and SYS (cross-NUMA) link types, and what each implies for P2P communication performance.
  4. NCCL's topology override mechanism: The NCCL_TOPO_FILE environment variable and the XML schema for describing GPU topology, including the <cpu>, <pci>, and <gpu> elements and their attributes.
  5. Sysfs and the Linux PCI subsystem: The /sys/bus/pci/devices/ hierarchy, the numa_node attribute, and how writing to it affects (or doesn't affect) higher-level software.
  6. The Proxmox virtualization stack: How hostpci mapping works, the Q35 machine type, and the relationship between the Proxmox mapping file (/etc/pve/mapping/pci.cfg) and the VM's virtual PCIe topology.
  7. The rc.local mechanism: How /etc/rc.local works as a boot-time script on systems that still support it (primarily Debian/Ubuntu with sysvinit compatibility).

Output Knowledge Created

This message creates several lasting artifacts and insights:

  1. A persistent NUMA affinity fix: The /etc/rc.local script ensures that every time the VM boots, the GPU PCI devices and their parent root ports are assigned the correct NUMA node numbers. This is a tangible improvement that survives reboots.
  2. A documented limitation: The message clearly establishes that nvidia-smi topo cannot be overridden by sysfs manipulation, which is valuable knowledge for anyone attempting similar virtualization optimizations. This saves future effort chasing a dead end.
  3. A combined workaround strategy: The integration of sysfs NUMA affinity + NCCL topology XML provides a multi-layered approximation of the correct topology. While neither fix alone is sufficient, together they address both memory allocation (sysfs) and communication planning (NCCL XML).
  4. A template for similar situations: The approach of identifying which software layer reads which topology information, and applying fixes at the appropriate layers, is a reusable pattern for performance debugging in virtualized GPU environments.

The Thinking Process Revealed

The assistant's reasoning in this message reveals a methodical, layered debugging approach. The thought process can be reconstructed as follows:

Step 1: Observe the failure. The nvidia-smi topo -m output still shows all PHB with NUMA Affinity: 0-1. The sysfs changes had no visible effect on the topology report.

Step 2: Diagnose the mechanism. Why didn't it work? Because nvidia-smi topo uses its own detection logic based on the PCIe tree structure (which root port, which bridge), not the sysfs numa_node attribute. The Q35 chipset presents all GPUs as being behind root ports on the same root complex (bus 0x00), so they all appear as PHB peers.

Step 3: Re-evaluate the strategy. The sysfs fix isn't worthless—it just works at a different layer. CUDA memory allocation reads sysfs numa_node, so the fix helps with memory affinity even if it doesn't affect the topology classification. The NCCL topo XML works at yet another layer (NCCL's internal topology detection). Together, they cover two of the three relevant layers (CUDA memory and NCCL communication), even though the driver layer (nvidia-smi) remains unaffected.

Step 4: Make it persistent. Since sysfs attributes are ephemeral (reset on reboot), the fix must be reapplied automatically. The rc.local mechanism is the simplest option.

Step 5: Execute. Write the rc.local script with the correct device paths and NUMA assignments, make it executable, and verify.

The assistant also shows awareness of the limits of this approach. It doesn't claim that the combined fix will restore bare-metal performance. It frames it as "our best bet"—an honest assessment that acknowledges the constraints while still pursuing the best available option.

The Broader Significance

This message encapsulates a fundamental challenge in virtualizing high-performance GPU workloads: the tension between the hypervisor's need to abstract hardware and the GPU's need for direct hardware access. Modern GPU frameworks (CUDA, NCCL, TensorRT) are deeply optimized for specific hardware topologies, and virtualization introduces layers of indirection that these frameworks cannot see through.

The sysfs numa_node manipulation is, in a sense, a form of topology forgery—telling the software stack that the hardware topology is something it isn't. It's a hack, but an informed one. The assistant understands exactly which software component reads which piece of topology information, and provides the "correct" answer to each component at the layer where it reads it. CUDA gets the right NUMA node from sysfs. NCCL gets the right topology from the XML file. The NVIDIA driver gets... the virtualized PCIe tree, and there's nothing to be done about that.

This layered approach to topology approximation is a valuable pattern for anyone working with virtualized GPU infrastructure. It won't match bare-metal P2P performance, but it can meaningfully reduce the gap by ensuring that memory allocation and communication planning make the best possible decisions given the constraints.

The message also demonstrates an important engineering virtue: knowing when to stop pursuing the ideal solution and instead optimize within constraints. The team had spent considerable effort trying to enable true P2P DMA through kernel modifications, chipset migration, and BIOS tweaks. The hardware topology (one GPU per root complex) was a fundamental constraint that no amount of software could overcome. The assistant's pivot to the sysfs + NCCL XML approach represents a mature acceptance of this reality and a focus on making the best of what is possible.