The Todo That Closed a Chapter: How One Status Update Captured the Limits of Virtualized GPU P2P

Introduction

In the middle of a grueling multi-session effort to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there is a message that, at first glance, appears to be nothing more than a routine status update. Message [msg 397] contains only a single tool call — a todowrite invocation that updates a JSON array of tasks, marking four high-priority items as "completed." The message is terse, almost mechanical: a structured data dump with no explanatory prose, no bash commands, no analysis. Yet this seemingly trivial update represents the quiet closing of a major investigative chapter. It is the moment when the assistant acknowledged that the fundamental goal of enabling Peer-to-Peer (P2P) DMA between GPUs inside a Proxmox virtual machine had been definitively abandoned — not because of a configuration error or a missing kernel parameter, but because of an immutable hardware topology constraint.

The Message in Full

The message reads:

[todowrite] {"todos":[
  {"content":"Investigate Proxmox host IOMMU/P2P configuration","priority":"high","status":"completed"},
  {"content":"Enable IOMMU on Proxmox host kernel cmdline","priority":"high","status":"completed"},
  {"content":"Fix BAR allocation for 8 GPUs on Q35 + pcie=1","priority":"high","status":"completed"},
  {"content":"Fix GPU NUMA topology in VM to match bare metal","priority":"high","status":"completed"},
  {...}
]}

The truncated array continues with additional tasks, but these four completed items tell the story. Each one represents hours of investigation, experimentation, and debugging across multiple machines — the Proxmox hypervisor host and the Ubuntu VM guest. Each one was a necessary step toward the goal of making eight GPUs communicate efficiently for distributed model inference.

Why This Message Was Written

To understand why this message exists, one must understand the context that led to it. The session had been attempting to deploy a massive language model (GLM-5-NVFP4) across eight RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism. Early benchmarking in [msg 382] revealed a critical performance bottleneck: cross-GPU communication latency was approximately 13 microseconds for small transfers — an order of magnitude worse than what bare-metal P2P DMA would achieve. This latency directly impacted the model's throughput because tensor-parallel inference requires frequent all-reduce operations between GPUs.

The root cause was that inside the Proxmox VM, nvidia-smi topo -m showed every GPU-to-GPU link as PHB (PCIe Host Bridge) — the worst possible topology classification — with NS (Not Supported) for P2P DMA. On bare metal, these same GPUs would show NODE for GPUs on the same NUMA node and SYS for cross-NUMA links, with P2P fully supported. The virtualization layer was stripping away the hardware topology information and preventing direct GPU-to-GPU communication.

The assistant had been systematically working through a stack of potential fixes, each one peeling back a layer of the problem. The four completed tasks in message [msg 397] represent the final, conclusive round of investigation that proved the problem was unsolvable within the constraints of the current hardware and virtualization setup.

How Decisions Were Made

The decision-making process visible in the preceding messages reveals a methodical, layered approach. Each task was tackled in sequence, with the assistant testing hypotheses and discarding dead ends.

Task 1: Investigate Proxmox host IOMMU/P2P configuration. This was the reconnaissance phase. The assistant examined the host's PCIe topology using lspci -tvv and discovered that each of the eight GPUs was attached to its own dedicated PCIe root complex on the ASUS ESC8000A-E13 motherboard. This is an AMD EPYC platform where each GPU gets a direct x16 link to the CPU, with no intermediate PCIe switch. The IOMMU groups on the host confirmed this isolation — each GPU was in its own IOMMU group, meaning VFIO (the kernel driver used for PCIe passthrough in KVM) could not grant them permission to perform direct DMA to each other.

Task 2: Enable IOMMU on Proxmox host kernel cmdline. The assistant modified the Proxmox host's kernel boot parameters to include amd_iommu=on iommu=pt. The iommu=pt (pass-through) mode reduces translation overhead by allowing devices that are directly assigned to a VM to bypass some IOMMU translation steps. This was a necessary precondition for any P2P attempt but ultimately insufficient on its own.

Task 3: Fix BAR allocation for 8 GPUs on Q35 + pcie=1. This was a major hurdle. When the assistant migrated the VM from the legacy i440FX chipset to the modern Q35 chipset (required for proper PCIe passthrough), six of the eight GPUs disappeared from the guest. The root cause was a Base Address Register (BAR) allocation failure — the Q35 machine's PCIe topology didn't have enough address space for all eight GPUs' BARs. The fix was adding pci=realloc to the guest kernel command line, which forced the kernel to reassign BAR addresses. This was a significant achievement: all eight GPUs were now visible inside the VM with their full 96GB of VRAM.

Task 4: Fix GPU NUMA topology in VM to match bare metal. This was the assistant's creative workaround for the topology problem. Since the VM couldn't expose the real NUMA topology through the virtual PCIe hierarchy, the assistant attempted two approaches. First, it created an NCCL topology XML file (~/nccl_topo.xml) that manually specified which GPUs belonged to which NUMA node, bypassing the kernel's topology detection. Second, it discovered that the numa_node sysfs attribute for PCI devices could be written directly, and set all eight GPUs and their parent PCIe root ports to the correct NUMA affinity. This was made persistent via /etc/rc.local.

Assumptions Made

Several assumptions underpinned this work, some of which proved incorrect:

The assumption that ACS (Access Control Services) was the culprit. Early in the investigation, the assistant hypothesized that ACS was blocking P2P transactions between GPUs in different IOMMU groups. This led to an attempt to disable ACS in the host BIOS, which required rebooting the Proxmox host. The ACS disable did renumber the IOMMU groups but critically failed to merge them — conclusively proving that ACS was not the cause of the isolation. The GPUs remained in separate groups because each one had its own root complex, not because ACS was enforcing access controls.

The assumption that software workarounds could fully compensate. The NCCL topology XML and sysfs NUMA affinity hacks were clever workarounds, but the assistant discovered they had limited effect. nvidia-smi topo -m continued to show PHB for all GPU pairs because NVIDIA's topology detection reads the actual PCIe hierarchy, not the sysfs numa_node attribute. The NCCL topology file helps NCCL choose better communication algorithms, but it cannot change the fundamental fact that P2P DMA is unsupported.

The assumption that QEMU's pxb-pcie expander buses could fix the topology. The assistant briefly explored using custom QEMU arguments to create PCIe expander buses with NUMA affinity, which would have made the virtual topology match the physical one. This was abandoned because it conflicted with Proxmox's automatic hostpci device management — Proxmox doesn't support pxb-pcie through its configuration syntax, and mixing manual -device arguments with auto-generated ones would cause conflicts.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption was that the P2P problem was solvable through software configuration alone. The investigation revealed a fundamental hardware constraint: the ASUS ESC8000A-E13 motherboard, designed for maximum per-GPU bandwidth, places each GPU on its own PCIe root complex. There is no shared PCIe switch that would allow VFIO to present the GPUs as a single IOMMU group. This is a deliberate design choice — each GPU gets a dedicated x16 link to the CPU without sharing bandwidth — but it has the side effect of making P2P DMA impossible in a virtualized environment.

The assistant also initially assumed that ACS disable would be the key to merging IOMMU groups, based on prior experience with other platforms where ACS was the blocking factor. This was a reasonable hypothesis but turned out to be wrong for this specific hardware topology.

Input Knowledge Required

To understand message [msg 397], the reader needs knowledge of several domains:

Output Knowledge Created

Message [msg 397] creates a definitive status checkpoint. It marks the completion of all investigative work into the P2P/IOMMU/PCIe topology problem. The output knowledge is:

  1. The P2P problem is conclusively diagnosed: The root cause is hardware topology (one GPU per root complex), not software configuration (ACS, IOMMU settings, or kernel parameters).
  2. A stable VM configuration exists: All eight GPUs are visible with full VRAM, BAR allocation is fixed, and NUMA affinity is set via sysfs and NCCL topology file.
  3. The remaining limitations are documented: P2P remains NS, nvidia-smi topo still shows PHB everywhere, and the ~13µs latency floor is inherent.
  4. A decision point is reached: The assistant can proceed to benchmarking with the current best-effort configuration, or explore alternative approaches like expert parallelism (which reduces cross-GPU communication) or bare-metal deployment.

The Thinking Process

The thinking process visible in the messages leading up to [msg 397] shows a pattern of systematic elimination. The assistant starts with the most promising hypothesis (ACS is blocking P2P), tests it (ACS disable), and when it fails, moves to the next layer (sysfs NUMA affinity, NCCL topology XML). Each failure provides information that narrows the search space.

What's notable is the assistant's willingness to try unconventional approaches. Writing directly to /sys/bus/pci/devices/*/numa_node is not a standard configuration technique — it's a low-level sysfs manipulation that most administrators wouldn't think to attempt. The assistant discovered this capability and used it effectively, even though it ultimately couldn't overcome the hardware limitation.

The todo update in [msg 397] is the moment when the assistant steps back and takes stock. The four completed tasks represent a complete cycle of investigation: understand the problem, attempt a fix, verify the result, and document the outcome. The truncated array suggests there are more tasks remaining — likely related to benchmarking and performance tuning — but the P2P chapter is definitively closed.

Conclusion

Message [msg 397] is a milestone marker in a complex engineering investigation. It captures the moment when the assistant acknowledged the limits of what could be achieved through software configuration in a virtualized environment. The four completed tasks represent hours of careful work: host kernel modifications, VM chipset migration, BAR allocation debugging, and creative sysfs manipulation. Each task was completed successfully in the sense that it was executed correctly — the IOMMU is enabled, the BARs are allocated, the NUMA affinity is set. Yet the ultimate goal of enabling P2P DMA remained out of reach, blocked not by a configuration error but by the physical arrangement of silicon on the motherboard.

This message is a testament to the value of systematic debugging. By methodically testing each hypothesis and documenting the results, the assistant built a complete picture of the system's constraints. The todo update is not an admission of defeat — it's a strategic pivot, clearing the board for the next phase of the work.