The PCIe Topology Ceiling: Confronting the Limits of GPU Peer-to-Peer DMA in Virtualized Environments

Introduction

In the high-stakes world of multi-GPU machine learning inference, every microsecond counts. When deploying large language models like GLM-5-NVFP4 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the communication fabric between GPUs becomes as critical as the compute itself. This article examines a pivotal moment in a technical conversation where an AI assistant, after successfully getting all eight GPUs detected and operational in a Proxmox virtual machine, confronts a deeper and more stubborn problem: the fundamental impossibility of enabling true Peer-to-Peer (P2P) DMA between GPUs in a virtualized environment.

The message under analysis—message index 367 in the conversation—represents a turning point. It is the moment where the team shifts from believing that P2P is a configuration issue that can be solved with the right settings, to understanding that it is a hardware topology constraint that no amount of kernel parameter tuning can fully overcome. This message synthesizes research, explains the underlying PCIe architecture, proposes two divergent paths forward, and delivers a sobering reality check through empirical measurement.

The Context: A Long Road to Eight GPUs

To understand why message 367 carries such weight, we must appreciate the journey that preceded it. The conversation began with setting up an ML environment on Ubuntu 24.04 with multiple GPUs, resolving complex build issues for flash-attn, and deploying the GLM-5-NVFP4 model using SGLang. The team then spent considerable effort debugging persistent NaN crashes during decode, which were eventually resolved by selecting working NSA backends.

The critical turn came when the team identified that cross-GPU communication was a major performance bottleneck. The Proxmox virtual machine environment introduced virtualization overhead that degraded the tightly-coupled GPU inference workload. The assistant embarked on a systematic investigation of the host's PCIe topology, modified Proxmox host kernel parameters to enable IOMMU passthrough (amd_iommu=on iommu=pt), migrated the VM from the legacy i440FX chipset to Q35 with proper PCIe device passthrough (pcie=1), and fixed a severe BAR allocation failure that initially prevented 6 of 8 GPUs from being detected.

The pci=realloc kernel parameter, applied to the guest after discovering that a cloud-init configuration file was overriding GRUB settings, finally allowed all eight GPUs to be recognized with their full 96GB VRAM each. The topology output showed PHB (PCIe Host Bridge) connections instead of the flat bus of the old i440FX configuration—a genuine improvement. But the nvidia-smi topo -p2p r command revealed a stark reality: every GPU pair showed NS (Not Supported) for P2P DMA.

The user's response was direct: "no bench, first can we fix p2p properly?" This set the stage for message 367.

The Message: A Research-Backed Reality Check

The assistant's response begins with a phrase that signals a shift in understanding: "This is very illuminating. The NVIDIA documentation explains exactly what we need. Here's the reality." The assistant had conducted web research, consulting NVIDIA's official AI Enterprise documentation on VM configuration, GitHub issues about NCCL P2P in virtual machines, and Linux kernel PCI subsystem patches. The message that follows is not a speculative guess—it is a synthesis of authoritative sources applied to the specific hardware topology of the ASUS ESC8000A-E13 motherboard with AMD EPYC Turin processors.

Let us quote the core analysis directly:

P2P between GPUs in a KVM VM is fundamentally limited. The NS (Not Supported) status is expected behavior when GPUs are on separate PCIe root ports without a physical PCIe switch connecting them.

This opening sentence is crucial. It reframes the problem: NS is not a bug or a misconfiguration—it is expected behavior. The assistant then elaborates with four key points that build a complete picture of the constraint.

Point 1: PCIe Hierarchy Domains

PCIe P2P requires devices to be in the same PCIe hierarchy domain — the PCIe spec doesn't require root complexes to route P2P transactions between hierarchy domains. On bare metal, your 8 GPUs are each on separate AMD Turin root complexes, so even bare metal wouldn't have true PCIe P2P (it would go through CPU Infinity Fabric instead).

This is a critical insight. The assistant correctly identifies that the problem is not caused by virtualization—it is merely exposed by it. On bare metal, the eight GPUs are each connected to their own dedicated PCIe root complex on the AMD EPYC platform. The motherboard design prioritizes maximum per-GPU bandwidth (each GPU gets a direct Gen5 x16 link to its own root complex) over the ability for GPUs to communicate directly with each other. Even without a hypervisor, P2P transactions between these GPUs would traverse the CPU's Infinity Fabric rather than occurring through a shared PCIe switch. The NS status in the VM reflects this underlying hardware reality.

Point 2: VFIO Adds Another Layer

In a VM, VFIO adds another layer — even when devices are behind virtual PCIe switches, VFIO must mediate DMA between devices in different IOMMU groups.

The assistant explains that even if one could construct an elaborate virtual PCIe topology inside the VM, the VFIO subsystem on the host must mediate DMA between devices in different IOMMU groups. The earlier investigation had already confirmed that each GPU resides in its own IOMMU group because each is on its own PCIe root complex. Disabling ACS (Access Control Services) in the BIOS had renumbered the groups but failed to merge them, conclusively proving that the isolation is topological, not policy-based.

Point 3: NVIDIA's Official Solution

NVIDIA's official solution (from their AI Enterprise docs) is to create a virtual PCIe topology with pcie-switch-upstream-port and pcie-switch-downstream-port devices, grouping GPUs behind virtual PCIe switches. This tricks the NVIDIA driver into seeing GPUs as being behind the same switch. However, even this doesn't create real hardware P2P — the actual data still goes through the host.

The assistant presents NVIDIA's recommended approach but immediately qualifies it with a crucial caveat: it is a "trick" that changes what the NVIDIA driver reports (changing NS to PIX in the topology output), but it does not change the physical reality of where data travels. This honest assessment prevents the team from pursuing an elaborate configuration that would yield no real performance benefit.

Point 4: The Real Question

The real question is whether NCCL can use SHM (shared memory) transport efficiently instead of P2P. With iommu=pt on the host, DMA operations should be faster (no IOMMU translation overhead), even without P2P.

This reframes the entire objective. Instead of chasing the unattainable goal of hardware P2P, the assistant suggests optimizing the fallback path that NCCL will use when P2P is unavailable: shared memory transport. The iommu=pt (passthrough) mode already applied to the host kernel eliminates IOMMU translation overhead for DMA operations, which should benefit the SHM path.

The Two Paths Forward

Having established the fundamental constraint, the assistant presents two distinct paths, each with different tradeoffs.

Path A: Virtual PCIe Switch Topology

This approach involves editing the Proxmox VM configuration with custom QEMU arguments to create virtual PCIe switch devices (pcie-switch-upstream-port and pcie-switch-downstream-port). The assistant provides a sketch of what this would look like:

qm set 128 -args '-device pcie-root-port,id=rp1,bus=pcie.0,slot=0x10,chassis=1 \
  -device pcie-pci-bridge,id=sw1,bus=rp1 ...'

However, the assistant immediately notes a practical obstacle: "this conflicts with how Proxmox manages hostpci devices." Proxmox's VM configuration system is not designed for manual PCIe topology manipulation of this complexity. The virtual switch approach is more feasible with libvirt-managed QEMU instances, where XML configuration provides finer-grained control. Even if implemented, as noted earlier, it would only change the reported topology without creating real hardware P2P.

Path B: Skip P2P, Focus on NCCL SHM Optimization

This path accepts the NS status and optimizes the communication that actually happens. The assistant argues that with iommu=pt already active on the host, the SHM transport path should already be faster than it was before the configuration changes. The key insight is that NS does not mean "no communication"—it means NCCL falls back from GPU Direct P2P DMA to a shared memory transport that moves data through the host's memory system. The question is whether this fallback path provides adequate bandwidth for the inference workload.

The Empirical Test: Measuring What We Have

Rather than continuing to theorize, the assistant executes a practical bandwidth test directly on the VM. The test uses PyTorch's copy_ operation between GPU 0 and GPU 1—a simple but revealing benchmark that measures the actual achievable bandwidth for cross-GPU transfers.

The results are illuminating:

4MB GPU0->GPU1 copy: 33.67 GB/s (124.6 us/copy)
12KB GPU0->GPU1 copy: 0.94 GB/s (13.1 us/copy)

The 4MB transfer achieving 33.67 GB/s is actually quite respectable. For comparison, PCIe Gen5 x16 has a theoretical bandwidth of approximately 64 GB/s per direction. Achieving roughly half of that through the virtualized path, with the data traversing the host's memory system via SHM rather than direct GPU-to-GPU DMA, is a non-trivial result. The 12KB transfer at 0.94 GB/s (13.1 microseconds per copy) is more modest but still functional for small control messages.

These numbers provide a crucial baseline. They tell the team what they can expect from the current configuration without any further P2P hacking. For the GLM-5-NVFP4 inference workload, which uses tensor parallelism across all eight GPUs, the communication pattern involves both large tensor transfers (during forward pass activations) and small control messages. The 33.67 GB/s figure for 4MB transfers suggests that the SHM path, while not as fast as true P2P DMA would be, may still be adequate for the inference use case.

The Thinking Process Visible in the Message

Message 367 reveals a sophisticated reasoning process. The assistant does not simply report research findings—it synthesizes them into actionable guidance. Several thinking patterns are evident:

First-principles reasoning: The assistant starts from the PCIe specification's definition of hierarchy domains and works upward to understand why P2P fails, rather than treating the NS status as an opaque error code.

Honest constraint identification: The assistant clearly distinguishes between what is theoretically possible (constructing virtual PCIe switches) and what is practically useful (the virtual switches don't create real P2P). This prevents wasted effort.

Reframing the problem: Instead of treating NS as a problem to be solved, the assistant reframes it as a design choice to be evaluated. The question becomes not "how do we enable P2P?" but "is the non-P2P path fast enough for our workload?"

Empirical grounding: The assistant does not stop at theory but immediately runs a bandwidth test to measure the actual performance of the available communication path. This grounds the discussion in data rather than speculation.

Contextual awareness: The assistant understands the broader project goals (deploying GLM-5-NVFP4 for inference) and evaluates the P2P question in that context. The bandwidth numbers are not presented in isolation but as data points relevant to the inference workload.

Assumptions and Knowledge Required

To fully understand this message, the reader needs knowledge across several domains. An understanding of PCIe topology—root complexes, hierarchy domains, switches, and how BAR (Base Address Register) allocation works—is essential. Familiarity with KVM/QEMU virtualization, particularly VFIO device passthrough and IOMMU groups, is required to grasp why the VM adds an extra layer of mediation. Knowledge of NVIDIA's GPU driver architecture, including how nvidia-smi topo determines P2P capability and how NCCL selects transport methods, is necessary to interpret the NS status. Finally, understanding the communication patterns of tensor-parallel LLM inference helps evaluate whether the measured bandwidth is sufficient.

The assistant makes several assumptions that are worth noting. It assumes that the SHM fallback path is the primary alternative to P2P for NCCL in this configuration—which is correct for the current NVIDIA driver and NCCL versions. It assumes that iommu=pt on the host provides meaningful benefit for the SHM path—a reasonable assumption given that it eliminates IOMMU translation overhead for all DMA operations. It assumes that constructing virtual PCIe switches in Proxmox would conflict with the hostpci management system—a practical assumption based on Proxmox's architecture, though the exact interaction would need to be tested.

One potential limitation of the analysis is that the bandwidth test uses PyTorch's copy_ operation rather than NCCL's actual communication primitives. While copy_ gives a reasonable approximation of achievable bandwidth, NCCL's all-reduce and all-gather operations may have different performance characteristics due to their collective nature and algorithmic choices. The assistant implicitly assumes that the copy_ benchmark is a useful proxy, which is a reasonable first step but may not capture the full picture.

What This Message Achieves

Message 367 creates several important outputs. First, it establishes a shared understanding of why P2P is NS—not as a configuration failure but as a consequence of the hardware topology. This prevents the team from pursuing futile workarounds. Second, it provides two clearly defined paths forward with honest assessments of their costs and benefits. Third, it delivers empirical baseline data that can be used to evaluate whether further optimization is necessary. Fourth, it reframes the conversation from "fixing P2P" to "optimizing the available communication path," which is a more productive framing.

The message also implicitly establishes a decision point. The bandwidth test results—33.67 GB/s for 4MB transfers—are presented without explicit judgment, but the implication is clear: this may be good enough for the inference workload. The team can now decide whether to invest effort in the complex virtual PCIe switch approach or accept the current performance and move on to benchmarking the actual model.

Conclusion

Message 367 represents a moment of clarity in a complex technical investigation. It is the message where the team stops trying to force the hardware to behave differently and starts measuring what it actually delivers. The assistant's synthesis of PCIe topology principles, NVIDIA documentation, and empirical measurement provides a solid foundation for the next phase of the project. Whether the team pursues virtual PCIe switches or accepts the SHM path, they now do so with a clear understanding of the constraints they face.

The broader lesson is one that resonates across systems engineering: not every performance limitation is a configuration bug waiting to be fixed. Some limitations are architectural features of the hardware, and the skill lies in recognizing them, measuring their impact, and making informed tradeoffs. Message 367 is a textbook example of this kind of technical judgment—combining research, reasoning, and measurement to transform a frustrating dead end into a clear path forward.