The P2P Question: When Security Isolation Meets Hardware Topology

"pcie acs - wouldn't disabling permit p2p? or is it more hw level thing? I always want an 8 gpu vm and if not don't really care about security - I use vms for grouping, snapshots, etc. no untrusted tenants"

In a single, deceptively simple question, the user of this opencode session cut to the heart of a multi-hour debugging odyssey. Message [msg 403] arrived at a critical juncture: the assistant had just finished researching and compiling an exhaustive list of BIOS settings for the ASUS ESC8000A-E13 server (see [msg 402]), a document that included the recommendation that ACS (Access Control Services) remain Enabled for proper VFIO device isolation. The user, who had been wrestling with the frustrating reality that Peer-to-Peer (P2P) DMA between GPUs was non-functional inside their Proxmox VM, saw a potential escape hatch. If ACS was what the assistant said was "required for VFIO isolation," and if ACS was what created the separate IOMMU groups that prevented P2P, then surely disabling it would solve the problem. The question was both technically incisive and deeply pragmatic: "wouldn't disabling permit p2p?"

The Context: A Long Struggle Against Virtualization Overhead

To understand why this message matters, one must appreciate the journey that led to it. The user was running a demanding AI inference workload—the GLM-5-NVFP4 model deployed via SGLang on 8 NVIDIA RTX PRO 6000 Blackwell GPUs—inside a Proxmox virtual machine. The system had been painstakingly assembled over multiple sessions: NVIDIA drivers installed, CUDA Toolkit configured, flash-attn built against the correct PyTorch version after resolving memory exhaustion during compilation, and the SGLang server launched with tensor parallelism across all 8 GPUs.

But a persistent performance bottleneck had emerged. Cross-GPU communication—the lifeblood of tensor-parallel inference—was suffering from a ~13 microsecond latency floor. This was far higher than what bare-metal P2P DMA would deliver, and it directly impacted the throughput of the model. The root cause was clear: inside the VM, nvidia-smi topo -m showed all GPU links as PHB (PCIe Host Bridge), meaning every cross-GPU transfer had to go through the host CPU rather than happening directly between GPUs. P2P was NS (Not Supported).

The assistant had already attempted a multi-pronged fix. The Proxmox host kernel was modified with amd_iommu=on iommu=pt to reduce translation overhead. The VM was migrated from the legacy i440FX chipset to Q35 with proper PCIe passthrough (pcie=1). A BAR allocation failure that caused 6 of 8 GPUs to vanish was fixed with pci=realloc. GPU NUMA affinity was set via sysfs to match the physical topology, and an NCCL topology XML was crafted to tell NCCL the real GPU-to-NUMA mapping. Yet P2P remained stubbornly NS.

The critical discovery, documented in the chunk summaries, was that an attempt to disable ACS in the BIOS had already been made—and it had failed to merge the IOMMU groups. The GPUs remained in separate groups because each one was physically attached to its own dedicated PCIe root complex on the AMD EPYC platform. There was no shared PCIe switch. The isolation was baked into the silicon, not enforced by ACS policy.

What the User Was Really Asking

The user's message operates on multiple levels. On the surface, it's a technical question about the relationship between ACS, IOMMU groups, and P2P DMA. The user had absorbed the assistant's BIOS guide, which stated that ACS should be Enabled because it's "required for VFIO isolation." The user's reasoning chain was straightforward: ACS creates IOMMU group separation → IOMMU group separation prevents P2P → disabling ACS should remove the separation → P2P should then work.

But beneath this technical query lies a far more important statement of values: "I always want an 8 gpu vm and if not don't really care about security - I use vms for grouping, snapshots, etc. no untrusted tenants." This is the user declaring their priorities explicitly. They are not running a multi-tenant cloud environment. They are a single user using VMs as a convenience layer—for snapshot management, environment isolation, and hardware grouping. Security isolation between devices is not a requirement; performance is. If ACS is the gatekeeper preventing P2P, they are willing to tear down that gate.

This is a crucial moment of alignment. The assistant had been operating under the assumption that VFIO isolation was a hard requirement—hence the recommendation to keep ACS enabled. The user is now correcting that assumption, saying: "I don't need that protection. Give me the performance."

The Gap Between Assumption and Reality

The user's question contains an implicit assumption: that ACS is the primary mechanism preventing P2P, and that disabling it is a simple BIOS toggle that would resolve the issue. This assumption is understandable—it follows directly from the information the assistant had just provided. In the BIOS settings table (msg [msg 402]), ACS was listed under "Must-Have Settings" with the rationale "Required for VFIO isolation. You have this since GPUs are in separate IOMMU groups." A reasonable reader would conclude that ACS is the cause of the IOMMU group separation.

However, the earlier investigation (documented in chunk 1 of segment 3) had already proven this assumption incorrect. The ACS disable attempt had been executed, and while it successfully renumbered the IOMMU groups, it critically failed to merge them. The GPUs remained isolated because the ASUS ESC8000A-E13 motherboard places each GPU on its own dedicated PCIe root complex. There is no PCIe switch connecting multiple GPUs to a single root port. Each GPU gets its own direct lane to the CPU's PCIe fabric. This design maximizes per-GPU bandwidth—each card gets a full Gen5 x16 link—but it creates separate IOMMU domains that VFIO cannot bridge, regardless of ACS settings.

This is the fundamental constraint that the user's question, however sharp, could not overcome. ACS controls whether devices behind a shared PCIe switch can bypass each other's traffic through the switch. It does not, and cannot, merge devices that are on entirely separate root complexes. The hardware topology is the immutable layer beneath all the software-configurable layers.

The Knowledge Exchange

Input knowledge required to understand this message includes: familiarity with PCIe topology concepts (root complexes, switches, endpoints); an understanding of IOMMU groups and how VFIO uses them for device isolation; awareness of ACS (Access Control Services) as a PCIe feature for traffic isolation between devices; and the specific context of the ASUS ESC8000A-E13 motherboard's one-GPU-per-root-complex design. The user also needed to know that the assistant had already attempted ACS disablement—though notably, the user's question suggests they may not have fully absorbed that result, or they were hoping the assistant had missed something.

Output knowledge created by this message is significant. First, it clarifies the user's threat model and priorities: security isolation is explicitly not a concern, and performance is paramount. This reorients the entire investigation toward more aggressive, potentially unsafe workarounds. Second, it forces a precise articulation of why ACS disablement doesn't help—distinguishing between ACS as a switch-level feature and the root-complex-level isolation that is the real barrier. Third, it sets the stage for the assistant's subsequent exploration of "hacky workarounds" like vfio_iommu_type1.allow_unsafe_interrupts and nv_peer_mem, which would not have been appropriate to discuss under a security-first mindset.

The Thinking Process Revealed

The user's reasoning, visible in the structure of the message, is a model of pragmatic engineering thinking. They start with a hypothesis ("pcie acs - wouldn't disabling permit p2p?"), immediately qualify it with healthy skepticism ("or is it more hw level thing?"), and then provide the context that justifies why they're willing to pursue the hypothesis even if it's unsafe ("I always want an 8 gpu vm and if not don't really care about security"). This three-part structure—hypothesis, self-critique, justification—shows a user who is technically engaged, aware of the tradeoffs, and explicitly communicating their constraints to the assistant.

The question mark after "or is it more hw level thing" is particularly telling. The user suspects that the barrier might be deeper than a BIOS setting, but they're not certain. They're asking the assistant to confirm or refute their understanding. They want to know: is ACS the lock on the door, or is the door welded shut at the architectural level?

Why This Message Matters

Message [msg 403] is a turning point in the conversation. Before it, the assistant was operating within the constraints of standard VFIO security boundaries, recommending ACS be enabled and treating IOMMU group separation as a given. After it, the assistant pivots to investigating kernel parameters and modules that can forcibly bypass IOMMU group isolation—tools like vfio_iommu_type1.allow_unsafe_interrupts and the nv_peer_mem driver. These are not solutions that would be proposed in a multi-tenant environment, but they are exactly what the user is asking for.

The message also serves as a reality check. The user's hope that ACS disablement would unlock P2P is ultimately dashed by the hardware topology. But by asking the question, the user forces the conversation to confront the true nature of the constraint. The answer—"it's a hardware-level thing"—is disappointing but clarifying. It means that no amount of BIOS tweaking or kernel parameter fiddling will make P2P work in this VM on this motherboard. The only remaining options are hacky workarounds that partially mitigate the problem, or accepting the ~13µs latency floor.

In the end, this message captures the essence of systems engineering: the moment when a clever software fix meets an immovable hardware reality, and the engineer must decide whether to work around it, accept it, or change the hardware.