The Architecture of Impossibility: An 8-GPU Proxmox VM's Battle Against the Physics of PCIe Topology

Introduction

In the high-stakes world of large language model inference, the difference between a successful deployment and a frustrating bottleneck often lies not in the model architecture or the GPU compute capability, but in the invisible plumbing that connects the hardware together. When deploying the GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts language model—across eight NVIDIA RTX PRO 6000 Blackwell GPUs inside a Proxmox virtual machine, the team encountered precisely this kind of hidden bottleneck. The session documented in Segment 3 represents a pivotal turning point: the moment when the investigation shifted from software configuration to hardware topology, from "can we enable this feature?" to "can we work around a fundamental physical constraint?"

This article synthesizes the work across the entire segment, tracing the arc from discovery through diagnosis to the formulation of pragmatic workarounds. It examines how the assistant and user collaboratively peeled back the layers of virtualization—from NCCL tuning to IOMMU configuration to PCIe topology mapping—and ultimately confronted a hardware reality that no software change could overcome. The journey took them through kernel parameter modifications, VM chipset migrations, BAR allocation crises, cloud-init traps, ACS experiments, NUMA topology investigations, and finally into the dark corners of insecure kernel parameters and hacky workarounds.

The Discovery: A Bottleneck Revealed

The segment opens with the team in possession of a comprehensive status report that consolidated everything learned across hours of debugging, benchmarking, and profiling. The hardware configuration was formidable: eight RTX PRO 6000 Blackwell GPUs with SM120 architecture, an AMD EPYC 9335 CPU, 432GB of RAM, CUDA 12.8, NVIDIA driver 590.48.01, and SGLang compiled from source. The model—GLM-5-NVFP4 with DeepSeek Sparse Attention forcing NSA attention backends—was running, producing valid outputs, and appearing healthy.

But the critical discovery was buried in the P2P diagnostics: the machine was running inside a Proxmox KVM virtual machine, and peer-to-peer DMA between GPUs was completely disabled. Every GPU pair showed "NS" —"Not Supported"—for P2P read operations. Cross-GPU transfers that should happen at PCIe Gen5 x16 speeds (64 GB/s theoretical) were instead bouncing through host memory with a latency floor of 13.7 microseconds per small transfer. For a model with 78 layers of all-reduce operations, those microseconds compound into milliseconds of overhead per decode step.

The benchmark results told a sobering story: peak throughput of approximately 485 total tokens per second at 64 concurrent requests, with single-stream throughput stuck at a mere 11 tok/s. The targets—1,000+ total tok/s and 100+ tok/s single-stream—remained distant. GPU profiling showed 100% utilization but only 55% power draw, the classic signature of a latency-bound workload where the GPUs spend most of their time waiting for data from their peers rather than computing.

The user confirmed root access to the Proxmox host (named kpro6), and the assistant began investigating the host's IOMMU configuration. The kernel command line lacked IOMMU enablement parameters, but eight AMD IOMMU units were detected in the hardware. This was a promising sign: the hardware supported IOMMU, it just needed to be properly configured.

The Data That Changed Everything

What followed was a sequence of diagnostic commands that fundamentally shifted the team's understanding of the problem. The assistant could not SSH directly to the Proxmox host from its environment—the hostname kpro6 was not resolvable from the assistant's network. This forced a delegation workflow: the assistant provided commands for the user to run on the host and asked for the output to be pasted back.

The user's response was a raw paste of diagnostic command outputs that collectively painted a devastating picture of the hardware topology. The key findings were:

Eight GPUs, eight IOMMU groups: Each GPU was in its own IOMMU group (10, 28, 42, 61, 72, 90, 101, 117), with no two GPUs sharing a group.
Eight separate PCIe root complexes: The lspci -tv output revealed that each GPU connected to its own dedicated root complex on the AMD EPYC Turin platform. The pattern was consistent: each root complex contained a "Turin PCIe Dummy Host Bridge" and a single PCIe slot connecting directly to one GPU.
No PCIe switches: There were no shared PCIe switches anywhere in the topology. Each GPU had a dedicated lane to its own root complex, with no hardware mechanism for direct GPU-to-GPU communication at the PCIe level.
VFIO was properly configured: The vfio_pci, vfio_iommu_type1, and iommufd modules were all loaded and functioning. The passthrough setup was correct—it was the hardware that was the limitation.
VM configuration used i440FX: The VM (ID 128, named "llm-one") used the legacy i440FX chipset with eight hostpci entries using mapping=pro6000, but critically missing the pcie=1 flag that would present the GPUs as proper PCIe devices. This data was the moment of truth. The assistant's working hypothesis—that ACS override patches could merge IOMMU groups and enable P2P—was conclusively refuted. ACS (Access Control Services) is a feature of PCIe switches, and there were no switches in this topology. Each GPU was on its own root complex, and no software configuration could bridge that physical separation.

The Three-Phase Remediation Plan

The assistant synthesized this diagnostic data into a structured, actionable plan organized across three layers of the system stack:

Layer 1: The IOMMU Translation Tax

The assistant identified that while IOMMU was active on the host, the kernel command line lacked the iommu=pt (passthrough) parameter. Without this, the IOMMU operated in translation mode, meaning every DMA transaction—every GPU memory read or write—had to go through IOMMU translation tables rather than being passed through directly. This added latency to every GPU memory transaction.

The fix was straightforward: add amd_iommu=on iommu=pt to the Proxmox host's kernel command line by editing /etc/kernel/cmdline and running proxmox-boot-tool refresh. This was Phase 2 of the plan and required a host reboot.

Layer 2: The VM Configuration Mismatch

The assistant identified three specific issues with how the GPUs were presented to the VM:

Missing pcie=1 flag: Without this flag, the GPUs were presented as conventional PCI devices behind the i440FX bus, not as PCIe devices. This disabled PCIe-native features including ACS and ARI, and critically prevented the NVIDIA driver from seeing proper PCIe capabilities.
i440FX machine type: The legacy chipset did not model PCIe natively. All devices appeared on a flat PCI bus.
Missing x-vga=off: A best-practice omission for compute GPUs. Phase 1 of the plan addressed the most immediate issue: adding pcie=1 to each of the eight hostpci entries in the VM configuration. This was a low-risk change that required only a VM restart, not a host reboot. The assistant hypothesized that this alone could improve P2P status from NS to something better, as the NVIDIA driver would be able to negotiate PCIe capabilities properly.

Layer 3: The Physical Topology Constraint

The deepest layer was the one that could not be changed: each GPU sat on its own PCIe root complex of the AMD EPYC Turin processor. This topology is standard for high-end AMD EPYC server platforms like the ASUS ESC8000A-E13 motherboard. It maximizes per-GPU bandwidth by giving each GPU its own dedicated PCIe lane to the CPU, but it fundamentally prevents the kind of low-latency P2P communication that NVLink-enabled systems or systems with PCIe switches can achieve.

The assistant noted this matter-of-factly: "This is typical for 8-GPU non-NVLink server configs." There was no Phase 4 for this layer—it was a physical constraint that software alone could not overcome.

The BAR Allocation Crisis: When Eight GPUs Became Two

The team executed Phase 1 of the plan, migrating the VM from the legacy i440FX chipset to the modern Q35 chipset with pcie=1 enabled. This was a necessary step for proper PCIe device passthrough and a prerequisite for any hope of P2P enablement. But when the VM rebooted with its new configuration, only 2 of the 8 GPUs were detected by the NVIDIA driver.

The dmesg output told a painful story. Each RTX PRO 6000 GPU requires three Base Address Register (BAR) regions: BAR0 (64MB for control registers), BAR2 (a staggering 128GB for VRAM mapping), and BAR4 (32MB). With eight GPUs, the total BAR2 requirement alone was 1TB. The Q35 virtual chipset's 64-bit MMIO window, as configured by the SeaBIOS firmware, spanned approximately 1.5TB—theoretically sufficient, but the firmware's PCI resource allocator failed to place BAR2 within six of the eight PCIe bridge windows.

The error messages were unambiguous:

pci 0000:01:00.0: BAR 2 [mem size 0x2000000000 64bit pref]: can't assign; no space
pci 0000:01:00.0: BAR 2 [mem 0x390000000000-0x391fffffffff 64bit pref]: failed to assign

And the NVIDIA driver's response was equally clear:

NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR2 is 0M @ 0x0 (PCI:0000:01:00.0)
nvidia: probe of 0000:01:00.0 failed with error -1

Only GPUs at PCI addresses 04:00.0 and 08:00.0 survived—the last GPU on each of the two PCIe root complexes. The other six were invisible, their BAR2 regions left unassigned at address zero.

The Kernel's Own Diagnosis

Amidst the sea of error messages in the dmesg output, one line stood out as the single most valuable piece of diagnostic information in the entire crisis:

pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc

This was the Linux kernel's PCI subsystem telling the administrator exactly what to do. The pci=realloc kernel parameter instructs the kernel to perform a second pass at PCI resource allocation, attempting to reassign BARs that the firmware failed to place during initial boot. It is a well-known workaround for systems where the firmware's PCI resource allocator is insufficiently sophisticated to handle extreme configurations—precisely the situation here.

The assistant recognized this message as the critical clue. But the path from diagnosis to execution was not straightforward. The user had already attempted a host-side fix—setting mch.above_4g_mem_size=2T via QEMU arguments—and immediately expressed uncertainty about whether to undo it. The user also raised a prescient question about switching from SeaBIOS to OVMF (UEFI firmware), which might handle large BAR allocations more gracefully.

The Research Pivot: From Action to Understanding

At this point, the assistant did something that distinguishes mature debugging from guesswork: it paused. Rather than doubling down on the mch.above_4g_mem_size approach or immediately prescribing pci=realloc, the assistant recognized that it needed a deeper understanding of the problem space.

The assistant fetched Gerd Hoffmann's blog post on QEMU physical address space sizing—Hoffmann is a core QEMU maintainer, and his post is the authoritative source on how QEMU allocates the 64-bit MMIO aperture for the Q35 chipset. The assistant also consulted Proxmox forum threads about RTX 5090 passthrough failures (a structurally identical problem involving large-VRAM GPUs on dual EPYC platforms) and a tutorial on PCI BAR debugging.

This research phase transformed the assistant's understanding. The key insight was that the mch.above_4g_mem_size parameter, while valid, was not the right lever to pull. The problem was not that the MMIO window was too small—it was that the SeaBIOS firmware's allocation algorithm was failing to place BAR2 within bridge windows that were, in theory, large enough. The fix needed to happen at the guest kernel level, not at the QEMU host level.

The Three-Option Plan and the User's Decision

The assistant synthesized the research into a clear, prioritized plan:

Option 1: Undo the mch.above_4g_mem_size argument and use an alternative approach with host-phys-bits to let SeaBIOS auto-detect the correct MMIO window size.

Option 2: Add pci=realloc to the guest kernel command line via GRUB. This was the least invasive fix—no firmware changes, no disk conversion, no host modifications. The guest kernel itself had suggested this approach.

Option 3: Switch from SeaBIOS to OVMF (UEFI firmware) with an explicitly configured large MMIO aperture of 1.5TB. This was the most robust solution but required creating an EFI disk and potentially converting the boot disk from MBR to GPT—a risky and invasive procedure.

The user's response was a model of efficient communication: "apply 2 to the guest." Three words that selected an option, specified a target, and implicitly accepted the assistant's recommendation.

The Cloud-Init Trap

The assistant's first attempt to add pci=realloc followed the standard Ubuntu procedure: edit /etc/default/grub, run update-grub, reboot. But after reboot, the kernel command line showed no trace of the parameter. Only 2 GPUs remained visible.

The clue was hidden in the update-grub output, which showed two sourcing events:

Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50-cloudimg-settings.cfg'

The cloud-init configuration file, generated by the Ubuntu cloud image build process, was setting GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0" after the main GRUB file was sourced. Because GRUB's configuration system sources grub.d/ files in lexicographic order after the main file, the cloud-init override silently clobbered the assistant's carefully added pci=realloc parameter.

This is a classic example of what makes infrastructure debugging so challenging: the system's behavior is determined not by any single configuration file, but by the ordering of multiple files in a chain. The fix was to modify the cloud-init file directly, appending pci=realloc to its existing console parameters. After a second reboot and a tense period of network unavailability, all 8 GPUs came online with their full 96GB VRAM.

The Moment of Truth: Eight GPUs Online, P2P Still Dead

With all 8 GPUs detected and operational, the team ran the critical diagnostic: nvidia-smi topo -p2p r. The result was an 8×8 matrix where every off-diagonal cell read "NS" (Not Supported). The companion command nvidia-smi topo -m showed all connections as "PHB" (PCIe Host Bridge)—the weakest possible topology classification, indicating that each GPU was behind its own host bridge with no shared PCIe switch.

The dmesg output confirmed that pci=realloc was active and that no BAR assignment errors remained. The PCIe subsystem was clean. But the NVIDIA driver's topology assessment was unambiguous: direct GPU-to-GPU DMA was not possible.

This was the moment the investigation pivoted from "how do we fix this configuration" to "what is the fundamental constraint?" The assistant's initial instinct was to proceed to benchmarking—to measure whether the Q35 migration alone had improved throughput even without P2P. But the user intervened with a sharp redirection: "no bench, first can we fix p2p properly?" This single message reframed the entire session. The team would not accept a degraded baseline; they would confront the hardest problem first.

The ACS Experiment: A Well-Known Hack That Didn't Help

The next logical step was to investigate Access Control Services (ACS), a PCIe capability that enforces isolation between downstream devices. In many virtualized environments, ACS is the mechanism that keeps devices in separate IOMMU groups. Disabling ACS in the BIOS has been known to merge IOMMU groups, potentially enabling P2P.

The user entered the BIOS and disabled ACS. The system rebooted. The assistant had prepared a comprehensive validation playbook covering IOMMU groups, IOMMU passthrough mode, PCIe link speeds, power management settings, and the ultimate P2P test inside the VM.

The results were sobering. The ACS disable successfully renumbered the IOMMU groups, but it critically failed to merge them. Each GPU remained in its own separate group. The assistant's analysis explained why: ACS only controls forwarding on PCIe bridges and switches that have multiple downstream devices. Since each GPU was on its own dedicated root complex—with no shared bridge between any pair—ACS had nothing to control. The isolation was baked into the hardware topology at a deeper level than ACS could reach.

This was the turning point. The assistant concluded with brutal honesty: "P2P (direct GPU-to-GPU DMA) cannot be enabled in this VM setup." The hardware topology—one GPU per root complex—was a fundamental constraint that no software configuration could overcome.

The NUMA Topology Investigation

Even without P2P, the team recognized a second-order problem: the VM was not exposing proper NUMA topology to the GPUs. On bare metal, GPUs 0–3 were on NUMA node 0 and GPUs 4–7 on NUMA node 1, with NODE interconnects within each group and SYS across groups. In the VM, all eight GPUs showed PHB with no NUMA distinction. This meant NCCL (the NVIDIA Collective Communications Library) could not differentiate between intra-socket and cross-socket communication paths, potentially causing suboptimal algorithm selection.

The assistant embarked on a detailed mapping exercise, correlating the Proxmox PCI resource mapping to physical GPU PCI addresses and host NUMA nodes. The discovery was fortuitous: the two Q35 PCIe root port groups (00:10.x for hostpci0–3 and 00:1c.x for hostpci4–7) happened to align with the physical NUMA domains. VM GPUs 0–3 mapped to physical NUMA 1, and VM GPUs 4–7 mapped to physical NUMA 0. The hardware grouping was correct, but the VM was not exposing this grouping as NUMA affinity.

The assistant explored several approaches to fix this. The most ambitious was using QEMU's pxb-pcie expander bus device with explicit NUMA node assignments, which would create virtual PCIe switches that mirror the physical topology. This approach, however, conflicted with Proxmox's automatic hostpci device management and would require removing the standard configuration in favor of raw QEMU arguments—a risky operation on a production VM.

A simpler but ultimately futile approach was writing directly to the sysfs numa_node attribute of each GPU's PCI device. The assistant assigned GPUs 0–3 to numa_node=1 and GPUs 4–7 to numa_node=0, and the kernel accepted the writes. But when the verification came, nvidia-smi topo -m showed no change. The NVIDIA driver builds its topology from the actual PCIe hierarchy, not from the kernel's NUMA affinity hints. The sysfs writes changed where the kernel thinks the devices lived for memory allocation, but they did not change the PCIe topology that the NVIDIA driver saw.

The Pragmatic Pivot: NCCL Topology XML

With the hardware-level approaches exhausted, the assistant pivoted to a software-level workaround: an NCCL topology XML file. NCCL supports loading a custom topology description via the NCCL_TOPO_FILE environment variable, which overrides its auto-detected topology. The assistant created a file at ~/nccl_topo.xml that described the true physical layout:

GPUs 0–3 on NUMA node 1, with PCIe Gen5 x16 links
GPUs 4–7 on NUMA node 0, with PCIe Gen5 x16 links
gdr="0" (GPU Direct RDMA unavailable, an honest reflection that P2P was not working) This approach would not enable P2P DMA—the actual data path would still go GPU→host memory→GPU with the same ~13µs latency floor. But it would give NCCL accurate topology information for algorithm selection, potentially improving all-reduce performance by keeping traffic within NUMA domains where possible. The choice of this approach over the more invasive QEMU pxb-pcie method reflected a mature engineering judgment: reliability and maintainability often trump theoretical performance. A software workaround that could be deployed, tested, and rolled back without touching the hypervisor configuration was inherently safer than modifying QEMU command lines or kernel parameters.

The BIOS Question: A Final Frontier

At this point, the user asked a question that opened a new front: "Btw any settings in bios that are highly warranted to set? Asus E13 GPU server." This question recognized that the software-level workarounds might only be part of the solution, and that the foundation of performance lay in the host firmware configuration.

The assistant launched parallel web searches, targeting both the specific ASUS platform and general AMD EPYC BIOS guidance. The search queries covered NPS (NUMA Per Socket), IOMMU configuration, ACS (Access Control Services), Above 4G Decoding, SR-IOV, PCIe Relaxed Ordering, and Resizable BAR—all settings that could impact multi-GPU inference performance. The searches converged on the AMD Instinct Customer Acceptance Guide's BIOS Settings page, an authoritative source for EPYC platform configuration.

This investigation, while not producing immediate changes, validated the user's intuition that BIOS configuration matters and provided a documented baseline for future optimization.

The Desperate Turn: "Hacky/Insecure Ways"

After the ACS experiment failed and the hardware topology was confirmed as the fundamental barrier, the user asked a question that crystallized the entire session's trajectory: "Are there hacky/insecure ways to get SYS/NODE in kvm?"

This eight-word question is remarkable for what it reveals. The user had fully absorbed the assistant's explanation of the hardware topology constraint. They understood that proper configuration could not solve this problem. They were willing to sacrifice security guarantees for performance. And they knew exactly what they were looking for: SYS and NODE are the topology types reported by nvidia-smi topo -m, representing the ideal NUMA-aware connectivity that NCCL needs for optimal algorithm selection.

The assistant responded by researching VFIO kernel parameters like vfio_iommu_type1.allow_unsafe_interrupts, investigating whether IOMMUFD (a newer VFIO backend) handled P2P differently, and exploring NVIDIA's nv_peer_mem module. This research represented the final frontier of the investigation—the point at which the team was willing to explore the system's darkest corners, trading security for performance.

The allow_unsafe_interrupts parameter is a well-known escape hatch in the VFIO ecosystem. By default, the VFIO IOMMU type1 driver refuses to enable P2P DMA between devices in different IOMMU groups as a security measure. Setting allow_unsafe_interrupts=1 tells the VFIO driver to bypass this check, potentially allowing DMA between devices that are not in the same IOMMU group. However, the assistant recognized a critical caveat: this parameter only controls the VFIO driver's software policy. If the underlying hardware topology—separate root complexes with no shared PCIe switch—prevents the IOMMU hardware from mapping DMA transactions between devices, then even allow_unsafe_interrupts would not help. The IOMMU hardware itself would reject the cross-group DMA mapping.

The nv_peer_mem module, developed by NVIDIA, is a different kind of workaround. It enables direct P2P communication between NVIDIA GPUs over the PCIe fabric by using the GPU's internal DMA engine. In bare-metal environments, nv_peer_mem can enable P2P even when the standard VFIO path is blocked. But in a KVM virtualized environment, the module would need to be loaded on the host, not the guest, and would need to interact correctly with the VFIO driver's DMA mapping—a complex and fragile interaction that the assistant flagged as potentially unstable.

What Was Learned: The Architecture of Impossibility

The central finding of this investigation is that the hardware topology of the ASUS ESC8000A-E13 motherboard—one GPU per dedicated PCIe root complex—fundamentally prevents P2P DMA in a KVM virtualized environment. This is not a configuration problem, not a BIOS setting, not a kernel parameter that can be tuned. It is a physical design constraint.

The mechanism works as follows:

The AMD EPYC platform exposes multiple PCIe root complexes, each connected to the CPU's Data Fabric.
The ASUS motherboard routes each GPU slot to its own root complex, maximizing per-GPU bandwidth.
In KVM with VFIO passthrough, each root complex becomes a separate IOMMU group.
VFIO enforces DMA isolation between IOMMU groups as a security guarantee.
No software configuration—not ACS disable, not kernel parameters, not sysfs hacks—can override this isolation when the GPUs are on separate root complexes. The practical consequence is a ~13µs latency floor for all inter-GPU communication in the VM, compared to ~5–8µs on bare metal. This doubling of latency is catastrophic for the all-reduce operations that underpin tensor-parallel inference.

The Pragmatic Path Forward

Despite the impossibility of true P2P DMA, the team established a set of pragmatic workarounds:

NCCL topology XML: A software-level hint that tells NCCL the correct NUMA topology, enabling better algorithm selection even without hardware P2P.
NUMA-aware VM configuration: Ensuring the VM's CPU and memory topology matches the physical NUMA layout, minimizing cross-socket traffic.
TP4+PP2 parallelism: Using a hybrid of tensor parallelism (within NUMA domains) and pipeline parallelism (across domains) to minimize cross-socket communication.
BIOS optimization: Configuring NPS mode, IOMMU settings, and power management for optimal GPU performance.
Insecure kernel parameters (last resort): Exploring vfio_iommu_type1.allow_unsafe_interrupts and nv_peer_mem as potential escape hatches, with full awareness of the security and stability risks. These workarounds do not eliminate the virtualization overhead, but they make the best of a constrained situation. For the user's GLM-5-NVFP4 inference workload, the path to 1,000+ tokens per second likely requires either accepting the ~13µs latency floor or moving to bare metal.

Conclusion

Segment 3 of this opencode session is a masterclass in systematic investigation at the hardware-software boundary. The team moved from GRUB configuration to PCIe topology, from IOMMU groups to ACS settings, from sysfs hacks to NCCL XML files, testing hypotheses and falsifying them with clean, unambiguous evidence. Each layer peeled back revealed another constraint, until the fundamental hardware topology stood bare.

The lesson is both sobering and empowering: some problems cannot be solved with software. The ASUS ESC8000A-E13's one-GPU-per-root-complex design is a deliberate engineering choice that optimizes for bandwidth at the expense of P2P capability. In a virtualized environment, this choice creates an insurmountable barrier. But understanding why the barrier exists—tracing it from the physical PCIe layout through the IOMMU groups to the VFIO driver's isolation guarantees—is itself a form of progress. It transforms a frustrating mystery into a documented constraint, and it enables informed decisions about whether to accept the limitation, work around it, or change the hardware.

For anyone deploying multi-GPU inference in virtualized environments, this investigation provides a roadmap: check your PCIe topology first, understand your IOMMU groups, and accept that some hardware designs are fundamentally incompatible with virtualized P2P DMA. The architecture of impossibility is, paradoxically, a form of knowledge that enables progress.

References

[1] "The Virtualization Ceiling: How an 8-GPU LLM Deployment Collided with the Physics of PCIe Topology" — Chunk 0 article covering the discovery phase, host topology investigation, and three-phase remediation plan.

[2] "The Anatomy of a Recovery: From IOMMU Group Failure to PCI BAR Rescue in an 8-GPU Proxmox VM" — Chunk 1 article covering the ACS failure, BAR allocation crisis, pci=realloc fix, and cloud-init gotcha.

[3] "The Architecture of Impossibility: A Deep Dive into GPU P2P DMA Across Eight Blackwell GPUs in a Proxmox VM" — Chunk 2 article covering the deep dive into why P2P is impossible, NUMA topology investigation, NCCL XML workaround, and hacky workarounds.