Mapping the Virtual PCIe Topology: A Diagnostic Pivot in the Quest for GPU P2P in Proxmox

In the sprawling, multi-session effort to deploy a GLM-5-NVFP4 model across 8 RTX PRO 6000 Blackwell GPUs on a Proxmox VM, one of the most critical bottlenecks has been the inability to enable peer-to-peer (P2P) DMA between GPUs. After migrating the VM from the legacy i440FX chipset to Q35, fixing BAR allocation with pci=realloc, enabling IOMMU passthrough on the host, and even attempting to disable Access Control Services (ACS) in the BIOS, the team had reached a sobering conclusion: the hardware topology itself—each GPU on its own dedicated PCIe root complex on the ASUS ESC8000A-E13 motherboard—fundamentally prevents VFIO from granting direct P2P DMA access. The nvidia-smi topo matrix in the VM stubbornly showed PHB for every GPU pair, with no NUMA affinity distinction.

Then, in message [msg 370], the user presented a crucial data point: the bare-metal nvidia-smi topo output for the same machine. That output revealed a topology where GPUs 0–3 saw each other as NODE (same NUMA node), GPUs 4–7 saw each other as NODE, and cross-socket pairs showed SYS (crossing the Infinity Fabric interconnect). The assistant's analysis in [msg 371] confirmed that the VM topology was strictly worse: all GPU pairs showed PHB with no NUMA affinity at all, robbing NCCL of the ability to optimize communication locality. The user then provided the host-side NUMA mapping in [msg 372], showing which physical GPUs belonged to which NUMA node. Message [msg 374] is the next logical step in this diagnostic chain: understanding how the Proxmox resource mapping translates physical GPUs into the virtual PCIe topology, by examining the VM's PCIe tree.

The Message: A Quiet Diagnostic Step

The subject message is deceptively brief—a single bash command and its truncated output:

[assistant] Now I need to understand the mapping. The Proxmox hostpci uses resource mappings (mapping=pro6000). Let me check what order those map to physical GPUs. [bash] ssh 10.1.230.175 "lspci -tv 2>/dev/null | head -60" -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Device 1234:1111 +-10.0-[01]----00.0 NVIDIA Corporation Device 2bb5 +-10.1-[02]----00.0 NVIDIA Corporation Device 2bb5 +-10.2-[03]----00.0 NVIDIA Corporation Device 2bb5 +-10.3-[04]----00.0 NVIDIA Corporation Device 2bb5 +-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 +-1a.1 Intel Corporation 82801I (ICH9 Fa...

On its surface, this is a routine diagnostic command: run lspci -tv on the VM to visualize the PCIe device tree. But in the broader context of the investigation, this message represents a critical pivot. The assistant has accepted that true P2P DMA is impossible given the hardware topology, and has shifted focus to a more achievable goal: reconstructing proper NUMA affinity in the virtual topology so that NCCL can at least optimize communication patterns. To do that, the assistant needs to understand exactly how the Proxmox hostpci resource mapping assigns physical GPUs to virtual PCIe slots.

Why This Message Was Written

The reasoning behind this message is rooted in a specific gap in the assistant's knowledge. The assistant already knows, from [msg 372], that on the Proxmox host, GPUs at PCI addresses 01:00.0, 11:00.0, 61:00.0, and 71:00.0 belong to NUMA node 0, while 81:00.0, 91:00.0, e1:00.0, and f1:00.0 belong to NUMA node 1. It also knows, from [msg 373], that inside the VM, GPU indices 0–7 map to virtual PCI buses 01:00.0 through 08:00.0. But the critical missing piece is: which virtual bus corresponds to which physical GPU?

Proxmox's hostpci resource mapping system (configured via /etc/pve/mapping/pci/pro6000.cfg) determines the order in which physical GPUs are assigned to the VM's hostpci0 through hostpci7 slots. The assistant cannot directly read the Proxmox host's configuration files from inside the VM—it can only observe the effects of that configuration through the VM's PCIe tree. By running lspci -tv, the assistant hopes to see the virtual PCIe hierarchy and infer which root ports correspond to which hostpci mappings.

The specific motivation is twofold. First, the assistant needs to determine whether the first four hostpci slots (hostpci0–hostpci3) correspond to the four NUMA 0 GPUs or whether the mapping is interleaved. Second, understanding the virtual topology is a prerequisite for the next step: constructing custom QEMU arguments to create virtual PCIe expander buses and switches that would give the VM proper NUMA affinity, potentially changing the nvidia-smi topo output from uniform PHB to a topology with NODE and SYS distinctions that mirrors bare metal.

What the Output Reveals

The truncated lspci -tv output shows the first four GPUs connected via PCIe root ports at 00:10.0 through 00:10.3, each leading to a downstream bus ([01] through [04]) with an NVIDIA GPU at 00.0 on each bus. This is the expected topology for a Q35 chipset VM: the 00:10.x devices are PCIe root ports that provide direct PCIe hierarchy to passed-through devices. The output also shows the Intel 82G33 DRAM Controller (the Q35 host bridge) and some ICH9 devices (USB controllers), confirming the chipset configuration is correct.

Notably, the output is truncated at 60 lines, so the second group of GPUs (likely connected via 00:1c.0–00:1c.3, the ICH9 PCIe-to-PCI bridge ports) is not visible. This truncation is a practical limitation—the assistant used head -60 to avoid overwhelming output, but it means the full picture isn't available in this single message. The assistant will need to either run the command without the head limit or examine the mapping configuration directly on the host to complete the picture.

Assumptions and Their Implications

The assistant makes several assumptions in this message. The first is that the Proxmox resource mapping order directly determines the virtual PCIe bus numbering. This is a reasonable assumption—Proxmox's hostpci devices are typically assigned in order, with hostpci0 getting the first device in the mapping, hostpci1 the second, and so on. However, the exact mapping depends on the order of entries in the resource mapping file, which the assistant cannot see from inside the VM.

The second assumption is that lspci -tv will provide enough information to determine the physical-to-virtual mapping. While the PCIe tree shows which root port connects to which bus, it doesn't reveal which physical GPU (by host PCI address) is behind each virtual bus. The assistant would need to correlate the virtual bus addresses with the physical GPU UUIDs or serial numbers—information that requires cross-referencing with the Proxmox host.

The third, more subtle assumption is that reconstructing NUMA affinity in the VM will meaningfully improve performance. The assistant acknowledged in [msg 369] that the fundamental bottleneck is the ~13µs latency floor for small transfers, which is inherent to the GPU→host memory→GPU staged copy path via QEMU/KVM. Even with perfect NUMA affinity, the actual data path doesn't change—transfers still go through host memory. The improvement would come from NCCL's ability to optimize communication patterns (e.g., preferring intra-NUMA transfers in ring algorithms), but the magnitude of this improvement is uncertain.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of several domains. First, an understanding of Proxmox VE's device passthrough mechanism, particularly the hostpci configuration and resource mapping system. Second, familiarity with PCIe topology concepts: root complexes, root ports, bus/device/function addressing, and how lspci -tv represents the device tree. Third, awareness of the Q35 vs i440FX chipset distinction and how Q35 provides proper PCIe root ports for device passthrough. Fourth, knowledge of NUMA (Non-Uniform Memory Access) and how GPU-to-GPU communication latency varies depending on whether GPUs share a NUMA node or cross the inter-socket interconnect. Finally, the reader needs the context of the preceding investigation: the failed attempts to enable P2P, the ACS disable experiment, and the discovery that each GPU sits on its own root complex.

Output Knowledge Created

This message creates several pieces of knowledge. It confirms that the Q35 chipset is working correctly, with GPUs appearing behind proper PCIe root ports (00:10.0–00:10.3) rather than on a flat PCI bus. It establishes the virtual PCIe bus numbering for the first four GPUs (buses 01–04). It provides the foundation for the next diagnostic step: reading the Proxmox mapping configuration to determine which physical GPUs map to which virtual slots. And it sets the stage for the assistant's proposal in the following message ([msg 375]) to use custom QEMU arguments to build a more sophisticated virtual topology.

The Thinking Process

The assistant's reasoning in this message is methodical and layered. It begins with a clear statement of intent: "Now I need to understand the mapping." This signals a transition from the previous phase (diagnosing the P2P problem) to a new phase (understanding the virtual topology to reconstruct NUMA affinity). The assistant explicitly references the Proxmox resource mapping mechanism (mapping=pro6000), showing that it understands the abstraction layer between physical hardware and virtual devices.

The choice of lspci -tv is deliberate. The -t flag shows the PCIe tree in a hierarchical format, which is essential for understanding which root ports connect to which buses. The -v flag adds verbose information. The assistant pipes through head -60 to avoid flooding the conversation with the full tree, which would include non-GPU devices like USB controllers, SATA controllers, and possibly the virtual PCIe switch infrastructure.

The truncation of the output is itself revealing. The assistant sees the first four GPUs (buses 01–04) but not the remaining four (buses 05–08). This creates an information gap that the assistant will need to fill in subsequent messages. The assistant doesn't immediately re-run the command without the head limit—instead, it proceeds to the next logical step in [msg 375], asking the user to read the Proxmox mapping configuration file directly. This shows a preference for authoritative information (the mapping config) over inference from incomplete data.

Conclusion

Message [msg 374] is a quiet but essential diagnostic step in a complex investigation. It doesn't produce dramatic results or breakthrough insights. Instead, it does the unglamorous work of gathering data, establishing baselines, and setting up the next phase of analysis. The assistant's methodical approach—first understanding the problem (P2P impossible), then identifying the next-best improvement (NUMA affinity), then gathering the data needed to implement that improvement—is characteristic of effective troubleshooting in complex systems. The message reminds us that in systems engineering, progress often comes not from dramatic breakthroughs but from the patient, methodical accumulation of understanding, one lspci at a time.