The Desperate Question: "Are there hacky/insecure ways to get SYS/NODE in kvm?"

In the middle of a grueling debugging session spanning days, a user asks a question that marks a critical turning point in the conversation. The message is short—just eight words—but it carries enormous weight:

"Are there hacky/insecure ways to get SYS/NODE in kvm?"

This single question, found at index 415 in the conversation, represents the moment when the user abandons hope for a clean, supported solution to a fundamental hardware limitation and begins exploring the dark corners of kernel parameters and unsafe driver hacks. It is the pivot point between "how do I fix this properly" and "how do I make this work at any cost."

The Context: A Long Struggle with GPU P2P in Virtualization

To understand why this question is asked, we must trace the path that led here. The user is running a high-performance machine learning inference workload—the GLM-5-NVFP4 model deployed with SGLang—on 8 NVIDIA RTX PRO 6000 Blackwell GPUs inside a Proxmox virtual machine. The GPUs are physically connected to an ASUS ESC8000A-E13 motherboard with dual AMD EPYC 9335 processors, each GPU occupying its own dedicated PCIe Gen5 x16 slot.

The problem emerged during performance benchmarking in an earlier segment of the conversation (segment 2). Cross-GPU communication latency was measured at approximately 13 microseconds, far higher than the 5–8 microseconds expected for GPUs on the same NUMA node. The culprit: Peer-to-Peer (P2P) DMA was not functioning inside the VM. The NVIDIA driver reported NS (Not Supported) in the P2P matrix rather than the desired NODE or SYS topology types.

The assistant and user embarked on an extensive investigation. They modified the Proxmox host kernel command line to enable IOMMU passthrough (amd_iommu=on iommu=pt). They migrated the VM from the legacy i440FX chipset to Q35 with proper PCIe device passthrough. They fixed a severe BAR allocation failure that prevented 6 of 8 GPUs from being detected by adding pci=realloc to the guest kernel. They verified that all 8 GPUs were operational with their full 96GB VRAM and that PCIe Gen5 x16 links were active with ASPM disabled.

Then came the critical experiment: disabling ACS (Access Control Services) in the BIOS. The hypothesis was that ACS enforcement was creating separate IOMMU groups for each GPU, and disabling it would merge the GPUs into fewer groups, allowing VFIO to treat them as a single domain and thus enable P2P DMA. The user rebooted the host, entered the BIOS, and disabled ACS.

The result was telling—and disappointing. The IOMMU groups were renumbered, but each GPU remained in its own separate group. ACS disable had not merged them. The assistant explained why: each GPU sits on its own dedicated PCIe root complex on the AMD EPYC platform, with no shared PCIe switch between any pair of GPUs. ACS only controls forwarding on bridges and switches; when there is no shared bridge, ACS has nothing to do. The separate IOMMU groups are a direct consequence of the hardware topology—one GPU per root complex—not of ACS enforcement.

This revelation was captured in the assistant's message at index 414, which laid out the hardware topology in detail and concluded: "For your hardware, the ~13µs cross-GPU latency in the VM is fundamentally limited by separate IOMMU groups (due to separate root complexes, not ACS). VFIO can't allow cross-group P2P without explicit kernel support (which doesn't exist for this topology)."

The assistant offered a path forward focused on BIOS power settings, model parallelism strategies (TP4+PP2), NCCL tuning, and the ultimate fallback of bare metal. But these were all compromises—ways to live with the limitation, not ways to overcome it.

The Question Itself: What It Reveals

The user's response at index 415 cuts directly to the heart of the matter. They don't ask "Is there a supported way?" or "Can we reconfigure the hardware?" They ask for "hacky/insecure ways." This choice of words is deliberate and revealing.

First, it signals that the user has fully absorbed the assistant's explanation. They understand that the limitation is fundamental to the hardware topology and that no amount of proper configuration will enable P2P DMA across separate IOMMU groups in KVM. They are not asking for clarification or for alternative proper approaches—they are asking for workarounds.

Second, it reveals the user's value hierarchy. The phrase "hacky/insecure" acknowledges awareness that these approaches may compromise system stability, security, or both. Yet the user is willing to accept those risks. This is a user who prioritizes performance above all else. As they stated earlier in the conversation (message 403): "I always want an 8 gpu vm and if not don't really care about security - I use vms for grouping, snapshots, etc. no untrusted tenants." The VM is not a security boundary for them; it is an operational convenience for resource grouping and snapshot management.

Third, the specific terminology—"SYS/NODE"—demonstrates technical sophistication. These are the topology types reported by nvidia-smi topo -m. NODE indicates GPUs on the same NUMA node (connected via the CPU's Data Fabric), while SYS indicates GPUs across different NUMA nodes (connected via the system interconnect). The user knows exactly what they're looking for and what the expected values should be. They are not a novice asking vague questions; they are an experienced practitioner who has studied the NVIDIA topology output and understands what the ideal outcome looks like.

The Assumptions Embedded in the Question

The question carries several implicit assumptions worth examining.

The user assumes that "hacky/insecure ways" exist at all. This is not a given. The VFIO subsystem in the Linux kernel is designed with security as a primary concern, and the IOMMU group isolation that prevents cross-group P2P is a deliberate security feature. There may simply be no knob to turn, no parameter to set, no module to load that would override this behavior. The user's question assumes that somewhere in the kernel or driver stack, there is an escape hatch.

The user also assumes that the problem is solvable through software alone. They do not ask about hardware changes—different motherboards, PCIe switches, or NVLink-based platforms. They assume that the existing hardware can be made to work with the right software incantations. This assumption may be incorrect; the hardware topology may be an absolute constraint that no amount of kernel hacking can overcome.

There is also an assumption that the assistant knows about or can discover these hacks. The user is implicitly asking the assistant to serve as a research tool, to search the corners of the internet where these undocumented or dangerous techniques are discussed. This places the assistant in the role of a co-explorer of the system's dark edges.

The Thinking Process Visible in the Conversation

The user's question is brief, but the thinking behind it is visible when we examine the trajectory of the conversation. The user has been on a journey of progressive disillusionment. They started with the expectation that a modern server with 8 high-end GPUs would deliver full P2P performance in a VM. Each debugging step revealed another layer of constraint: first the ACS disable didn't work, then the root complex topology was identified as the true barrier, then the assistant explained that VFIO fundamentally cannot enable cross-group P2P for this topology.

At each step, the user has been processing this information and recalibrating their expectations. The question at index 415 represents the final recalibration: the acceptance that proper solutions are exhausted and the turn toward improper ones. The brevity of the question suggests a certain weariness—the user has been through a long session and is now cutting to the chase.

The Input Knowledge Required

To fully understand this question, one needs considerable background knowledge across multiple domains. One must understand PCIe topology concepts: root complexes, switches, bridges, and how they relate to IOMMU groups. One must understand VFIO and KVM virtualization: how device passthrough works, how IOMMU groups are formed, and why VFIO enforces group-level isolation. One must understand NVIDIA's GPU topology reporting: what NODE, SYS, PIX, PXB, and NS mean in the context of nvidia-smi topo -m. One must understand the AMD EPYC platform: the role of the Data Fabric (Infinity Fabric) in routing P2P traffic between root complexes. And one must understand the specific hardware configuration of the ASUS ESC8000A-E13 motherboard and its one-GPU-per-root-complex design.

Without this knowledge, the question would appear to be a simple request for a configuration trick. With this knowledge, it becomes a sophisticated acknowledgment of a fundamental hardware limitation and a request to explore the boundaries of what the kernel and drivers can be forced to do.

The Output Knowledge Created

This question generates the next phase of the investigation. In response (message 416), the assistant begins researching VFIO kernel parameters like vfio_iommu_type1.allow_unsafe_interrupts, searches for any allow_unsafe_p2p equivalent, and investigates NVIDIA's nv_peer_mem driver module. The question thus creates new knowledge about the limits of VFIO's configurability and the availability (or lack thereof) of unsafe P2P mechanisms.

More broadly, the question crystallizes the key insight of this entire segment of the conversation: that the hardware topology of the ASUS ESC8000A-E13 motherboard, while optimal for per-GPU bandwidth, creates an insurmountable barrier to P2P DMA in virtualized environments. The question and its answer serve as a documented data point for anyone attempting a similar configuration: if your GPUs are each on their own root complex, no amount of ACS manipulation, kernel parameter tuning, or unsafe driver hacks will enable P2P in KVM.

Conclusion

The user's question at index 415 is a masterclass in concise technical communication. In eight words, it communicates acceptance of a fundamental hardware limitation, willingness to sacrifice security for performance, deep technical knowledge of GPU topology types, and a request to explore the system's darkest corners. It is the turning point in a long debugging session, the moment when the conversation shifts from "how do we configure this properly" to "how do we make this work at any cost." For anyone studying the challenges of GPU virtualization on AMD EPYC platforms, this question—and the investigation it triggers—is a valuable case study in the hard limits of software abstraction over physical hardware topology.