When the Hacks Begin: A Pivot from Configuration to Exploitation in Pursuit of GPU P2P DMA in KVM

Introduction

In the course of deploying a 744-billion-parameter GLM-5-NVFP4 mixture-of-experts model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs inside a Proxmox virtual machine, a conversation reached a critical inflection point. After systematically exhausting every conventional configuration lever—BIOS settings, chipset migration, kernel parameters, PCIe reallocation, and Access Control Services (ACS) manipulation—the user and assistant found themselves confronting a fundamental hardware topology constraint that no amount of legitimate configuration could overcome. It was at this moment that the user asked a question that would redefine the trajectory of the investigation: "Are there hacky/insecure ways to get SYS/NODE in kvm?" ([msg 415]). The assistant's response, [msg 416], marks the pivot from conventional systems administration into the shadowy territory of unsafe kernel parameters, unmerged kernel patches, and community workarounds—a shift that reveals as much about the nature of GPU virtualization as it does about the determination of those who push its boundaries.

The Context: A Bottleneck Laid Bare

To understand the significance of this message, one must appreciate the journey that preceded it. The session had been an exhaustive multi-day effort to deploy and optimize inference serving for the GLM-5-NVFP4 model, a 744B MoE architecture with 256 experts and DeepSeek Sparse Attention. The hardware was formidable: 8 RTX PRO 6000 Blackwell GPUs (96GB VRAM each, 768GB total) connected via PCIe Gen5 x16 links to an AMD EPYC 9335 dual-socket server. The software stack included sglang, flash-attn, and PyTorch, all painstakingly built and configured.

The critical bottleneck that emerged was cross-GPU communication latency. Inside the VM, NVIDIA's P2P (peer-to-peer) matrix showed NS (Not Supported) between all GPU pairs, forcing NCCL to fall back to SHM (shared memory) transfers through host memory. The measured latency floor was approximately 13 microseconds per small transfer—compared to the 5-8 microseconds achievable on bare metal. For a model requiring massive all-reduce operations across 8 GPUs with every token, this overhead was crippling throughput.

The assistant had already executed a systematic campaign to address this. The Proxmox host kernel was modified with amd_iommu=on iommu=pt to enable IOMMU passthrough mode. The VM was migrated from the legacy i440FX chipset to Q35 with proper PCIe device passthrough. When the chipset migration caused a catastrophic BAR allocation failure that rendered 6 of 8 GPUs undetectable, the guest kernel was patched with pci=realloc. All 8 GPUs were restored, running at full PCIe Gen5 x16 speeds with ASPM disabled and the CPU governor set to performance mode.

The most significant intervention was disabling ACS (Access Control Services) in the BIOS—a setting that controls whether PCIe bridges forward peer-to-peer traffic between downstream devices. The theory was sound: ACS is precisely the mechanism that VFIO uses to enforce IOMMU group isolation. Disabling it should merge devices behind shared bridges into fewer IOMMU groups, potentially enabling VFIO to allow P2P DMA between GPUs in the same group.

But the theory collided with hardware reality. After the ACS disable, the IOMMU groups were renumbered but not merged. Each GPU remained in its own isolated group. The reason was fundamental to the ASUS ESC8000A-E13 motherboard design: each of the 8 GPUs is attached to its own dedicated PCIe root complex on the AMD EPYC platform. There are no shared PCIe switches or bridges between any pair of GPUs. ACS has nothing to enforce when there is no shared bridge—and conversely, disabling it cannot create a shared bridge where none exists. The separate IOMMU groups are a consequence of separate root complexes, not ACS enforcement.

The Message: A Research Pivot into Unsafe Territory

The assistant's response at [msg 416] is the direct answer to the user's question about hacky workarounds. The message is notable for what it reveals about the assistant's reasoning process and the state of knowledge at this juncture.

The message begins with a direct acknowledgment: "Yes, there are a few hacky approaches. Let me think through what could actually work." This opening is significant. The phrase "let me think through" signals that the assistant is reasoning in real-time, not retrieving a pre-packaged answer. The word "actually" is telling—it acknowledges that many proposed workarounds may be theoretical rather than practical.

The assistant then launches into "Approach 1: vfio_iommu_type1.allow_unsafe_interrupts + allow_unsafe_p2p (if it exists)." This is a reference to a VFIO kernel module parameter that relaxes safety checks around interrupt remapping. The assistant correctly notes that there was discussion about an equivalent allow_unsafe_p2p parameter but expresses uncertainty about whether it was ever merged into the kernel: "There was discussion about an allow_unsafe_p2p equivalent but I'm not sure it was ever merged. Let me check."

The message then dispatches two web searches. The first searches for vfio_iommu_type1 allow_unsafe_p2p peer-to-peer DMA between IOMMU groups KVM QEMU GPU. The second searches for NVIDIA GPU P2P KVM VFIO separate IOMMU groups hack workaround "peer_memdev" OR "nvidia-peermem" OR "nv_peer_mem" virtual machine. These searches reveal the assistant's research strategy: it is looking for both kernel-level parameters and NVIDIA-specific peer memory modules that might bypass the IOMMU isolation.

The Reasoning and Motivation

Why was this message written? The surface-level answer is straightforward: the user asked a direct question, and the assistant is answering it. But the deeper motivation is more interesting. This message represents the assistant's recognition that the conventional path has been exhausted.

Throughout the preceding conversation, the assistant had operated within the boundaries of supported configuration. Every suggestion—BIOS settings, kernel parameters, VM chipset migration, PCIe reallocation—was within the realm of standard systems administration. The assistant had even correctly diagnosed why ACS disable wouldn't work for this topology, explaining that "since each GPU is on its own root complex (not behind a shared bridge), there's no ACS enforcement point between them" ([msg 412]).

But the user's explicit request for "hacky/insecure ways" changed the frame. The user had already stated their position: "I always want an 8 gpu vm and if not don't really care about security - I use vms for grouping, snapshots, etc. no untrusted tenants" ([msg 403]). With this mandate, the assistant was authorized to explore territory that would normally be off-limits in a production environment.

The assistant's motivation in this message is to systematically survey the landscape of possible workarounds, even those that are experimental, unmerged, or dangerous. The message is essentially a research plan: identify the candidate approaches, verify their existence and viability through web searches, and then present the findings.

Assumptions and Potential Mistakes

Several assumptions underpin this message. The first is that allow_unsafe_p2p or an equivalent parameter might exist in the VFIO kernel module. The assistant is appropriately cautious about this, expressing uncertainty and proposing to verify through web search. This is a reasonable approach, but it reflects an assumption that the kernel community might have created such an escape hatch—an assumption that may or may not be correct.

The second assumption is that the NVIDIA nv_peer_mem (or nvidia-peermem) driver module might provide a path to P2P in a VM context. This module is designed for GPU-to-GPU direct memory access on bare metal systems with NVSwitch or NVLink. Whether it functions at all inside a VFIO-passthrough VM with separate IOMMU groups is highly uncertain.

A potential mistake in the reasoning is the implicit assumption that any software-level workaround can overcome a hardware topology constraint. The assistant had already established that each GPU is on its own PCIe root complex—a physical design decision by ASUS. The IOMMU groups are a consequence of this topology, not an arbitrary software policy. Even if kernel parameters are relaxed, the physical PCIe hierarchy may still prevent direct GPU-to-GPU DMA without traversing the CPU's Data Fabric, which in a VM context requires IOMMU mediation.

Input Knowledge Required

To understand this message, one needs substantial background in several domains. First, the PCIe topology of the AMD EPYC platform: the concept of root complexes, PCIe switches, and how IOMMU groups are derived from the physical hierarchy. Second, the VFIO (Virtual Function I/O) framework and its role in device passthrough for KVM: how IOMMU groups determine the granularity of device isolation and why cross-group DMA is blocked. Third, the specific history of the allow_unsafe_interrupts parameter and the community discussions around allow_unsafe_p2p. Fourth, NVIDIA's peer-to-peer DMA infrastructure, including nv_peer_mem and the NCCL library's topology detection. Fifth, the Proxmox VE environment and its PCI mapping system.

The user brings significant domain knowledge to this exchange. The question about ACS and P2P ([msg 403]) demonstrates familiarity with PCIe concepts. The follow-up question about root complex topology ([msg 413]) shows engagement with the hardware-level analysis. The request for "hacky/insecure ways" reflects a pragmatic willingness to trade security for performance.

Output Knowledge Created

This message creates several forms of knowledge. Most immediately, it establishes a research agenda for exploring VFIO kernel parameters and NVIDIA peer memory modules as potential workarounds. The web searches, when their results return, will provide concrete information about the existence and viability of these approaches.

More broadly, the message documents the boundary of conventional GPU virtualization tuning. It represents the point at which standard configuration ends and experimental workarounds begin. For anyone attempting to deploy multi-GPU inference workloads in virtualized environments, this message—and the conversation it belongs to—serves as a case study in the limitations of VFIO passthrough for tightly-coupled GPU workloads.

The message also implicitly creates knowledge about the ASUS ESC8000A-E13 platform. The discovery that ACS disable does not merge IOMMU groups for this motherboard is a specific hardware finding that would be valuable to anyone considering this platform for multi-GPU virtualization.

The Thinking Process Visible

The assistant's thinking process is unusually visible in this message. The phrase "Let me think through what could actually work" is a rare moment where the assistant explicitly signals its reasoning mode. The subsequent structure—identifying a candidate approach, expressing uncertainty about its existence, and proposing to verify—reveals a methodical research methodology.

The assistant is not simply retrieving a known answer. It is constructing a hypothesis ("there might be an allow_unsafe_p2p parameter") and then designing a test (the web search) to validate it. This is the thinking process of an engineer who knows the landscape well enough to guess where solutions might exist but is humble enough to verify before asserting.

The two web searches are also revealing. The first targets the specific kernel parameter question. The second is broader, searching for any hack or workaround involving NVIDIA peer memory in virtual machines. This two-pronged approach—specific and exploratory—is characteristic of effective technical research.

Conclusion

The message at [msg 416] is a turning point in a larger narrative about the challenges of GPU virtualization for large-scale inference. It represents the moment when the investigation moved from "how do we configure this correctly" to "how do we make this work at any cost." The assistant's willingness to explore unsafe territory, combined with its methodical approach to verification, reflects the tension at the heart of this conversation: the desire for maximum performance colliding with the architectural constraints of virtualized hardware.

Whether the hacky approaches will bear fruit remains to be seen. But the message itself is a valuable artifact—a snapshot of the reasoning process at the boundary of known solutions, where the only way forward is to question assumptions and explore the undocumented edges of the system.