The P2P Breakthrough: How Disabling HMM Unlocked Direct GPU Communication in an LXC Container

Introduction

In the course of deploying the GLM-5-NVFP4 large language model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, a critical performance bottleneck had been looming since the earliest stages of the project. The inference server required efficient peer-to-peer (P2P) communication between GPUs to handle tensor parallelism, but the team had been running inside a Proxmox KVM virtual machine where VFIO-based GPU passthrough imposed a severe topology limitation. Every attempt to enable direct GPU-to-GPU communication had been thwarted by the virtualization layer, resulting in a PHB (PCIe Host Bridge) topology where all GPU traffic had to route through host memory. The subject message at index 569 represents the triumphant culmination of this debugging saga — the moment when the team confirmed that a switch to LXC containers, combined with a critical kernel module fix, had finally delivered true bare-metal P2P performance.

The Message

The assistant wrote:

Excellent results! - All GPU pairs have P2P access (no VFIO barriers!) - Same-NUMA P2P: 53.76 GB/s (vs KVM which was limited by PHB topology) - Cross-NUMA P2P: 40.24 GB/s

>

This is a massive improvement. In the KVM VM, P2P was likely going through host memory due to VFIO, causing the ~13us latency floor. Now we have direct PCIe P2P.

>

Let me now launch sglang. First, let me check all the dependencies are in order.

This was followed by a todowrite tool call updating task statuses, marking the P2P verification as completed and preparing for the next phase: launching the SGLang inference server.

The Long Road to P2P

To understand why this message carries such weight, one must appreciate the journey that preceded it. The project had spanned five full segments of troubleshooting. In Segment 3, the team had investigated enabling P2P DMA within a Proxmox KVM VM by modifying host kernel parameters, migrating to the Q35 chipset, fixing BAR allocation, and attempting to disable ACS (Access Control Services). Despite extensive effort, the fundamental hardware topology — each GPU on its own PCIe root complex — made P2P impossible within the VM. The VFIO layer simply could not provide direct peer access between devices attached to different root complexes.

Segment 4 saw a pivot to LXC containers, which share the host kernel and thus offer a more direct path to hardware. However, this approach was immediately blocked by a CUDA initialization failure. The cuInit() call either hung indefinitely or returned error code 3 (CUDA_ERROR_NOT_INITIALIZED). This was a show-stopper that prevented any GPU compute whatsoever.

The breakthrough came in the moments just before the subject message. The assistant discovered a GitHub issue (NVIDIA/open-gpu-kernel-modules #947, closed as a duplicate of #797) describing the exact same problem. The root cause was the NVIDIA open kernel module's Heterogeneous Memory Management (HMM) feature, which was incompatible with the Proxmox VE kernel (6.8.12-9). The fix was elegantly simple: set uvm_disable_hmm=1 as a module parameter for nvidia_uvm, then reload the module. This immediately resolved the CUDA initialization failure, and the assistant confirmed all 8 GPUs were accessible from within the LXC container.

Why This Message Was Written

The subject message serves multiple purposes simultaneously. First and foremost, it is a status report — the assistant is communicating the results of the P2P bandwidth test that was initiated in the previous round. The tone is celebratory ("Excellent results!", "This is a massive improvement"), reflecting genuine progress after what had been a frustrating and protracted debugging process.

Second, it is a decision point. The assistant uses the confirmed P2P results as the green light to proceed with launching the SGLang inference server. The bandwidth numbers — 53.76 GB/s same-NUMA and 40.24 GB/s cross-NUMA — are compared favorably against the KVM baseline, establishing that the LXC approach has overcome the primary performance bottleneck. The implicit decision is: "P2P is good enough now, let's move forward with deployment."

Third, it is a knowledge consolidation message. The assistant explicitly contrasts the LXC topology (NODE/SYS, meaning direct PCIe connectivity within the same NUMA node, and PCIe fabric connectivity across NUMA nodes) with the KVM topology (PHB, meaning all traffic routed through the host bridge). This comparison crystallizes the understanding that the virtualization layer was the root cause of poor performance, not the GPU hardware or driver configuration.

Reasoning and Decision-Making

The reasoning visible in this message is both retrospective and prospective. Retrospectively, the assistant interprets the bandwidth numbers: "In the KVM VM, P2P was likely going through host memory due to VFIO, causing the ~13us latency floor." This is an inference based on the known behavior of VFIO — when GPUs cannot directly address each other's memory, the VFIO mediator must copy data through host memory, adding significant latency. The ~13 microsecond figure is referenced as a known baseline from earlier testing (not shown in the immediate context but established in prior segments).

Prospectively, the assistant decides to "launch sglang" and begins checking dependencies. This is a natural transition: with the infrastructure validated, the focus shifts back to the original goal of running the GLM-5-NVFP4 model. The todo list update formalizes this transition, moving P2P verification to "completed" and setting up new tasks for the deployment phase.

A subtle but important decision is also visible in what the assistant does not do. There is no attempt to further optimize P2P bandwidth, no investigation of why cross-NUMA bandwidth is lower than same-NUMA, and no exploration of NVLink or other interconnect technologies. The assistant implicitly judges that 53 GB/s is sufficient for the tensor parallelism requirements of the model and moves on. This is a pragmatic engineering decision — good enough is good enough when the goal is end-to-end inference deployment.

Assumptions and Potential Pitfalls

Several assumptions underpin this message, some of which warrant scrutiny. The assistant assumes that the P2P bandwidth of 53 GB/s same-NUMA and 40 GB/s cross-NUMA will translate directly into improved inference throughput. While P2P is certainly necessary for efficient tensor parallelism, it is not sufficient — the inference kernels themselves (FlashInfer, MoE dispatch, attention backends) must also be properly tuned for the Blackwell SM120 architecture. The assistant acknowledges this implicitly by planning to "check all the dependencies" before launching sglang.

The comparison to the KVM baseline is also somewhat speculative. The assistant states that in the KVM VM, "P2P was likely going through host memory due to VFIO, causing the ~13us latency floor." While this is a reasonable inference, the actual bottleneck in the KVM case could have been multifactorial — including the VFIO interrupt handling, IOMMU page table walks, or even the GPU driver's behavior under virtualization. The ~13 microsecond figure is referenced as a known quantity, but its precise origin is not re-established in this message.

Another assumption is that the uvm_disable_hmm=1 fix is stable and production-worthy. The assistant has only tested it with a quick Python script running matrix multiplication. There may be edge cases where disabling HMM causes other issues — for example, with Unified Memory access patterns, CUDA IPC, or multi-process service daemons. The assistant's decision to proceed without stress-testing the fix is a calculated risk.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, an understanding of GPU topology and the nvidia-smi topo -m output is essential — the distinction between NODE (direct PCIe fabric connection), SYS (through the CPU/system), and PHB (through the PCIe host bridge) determines whether P2P communication is efficient or must go through host memory. Second, familiarity with VFIO and KVM virtualization helps explain why the earlier approach failed: VFIO intercepts PCIe transactions and cannot provide the direct peer-to-peer DMA that GPUs require for high-bandwidth communication. Third, knowledge of the NVIDIA open kernel module's HMM feature and its compatibility with various Linux kernels is needed to understand why uvm_disable_hmm=1 was the fix. Finally, an appreciation for the inference serving architecture — tensor parallelism requires frequent all-reduce operations between GPUs, making P2P bandwidth a first-order performance factor — explains why this verification was so critical.

Output Knowledge Created

This message creates several pieces of valuable knowledge. It establishes a concrete bandwidth baseline for P2P communication on this specific hardware configuration: 53.76 GB/s same-NUMA and 40.24 GB/s cross-NUMA for 8 NVIDIA RTX PRO 6000 Blackwell GPUs in an LXC container on Proxmox VE. It confirms that the NODE/SYS topology (as opposed to PHB) is achievable in a containerized environment, providing a reference architecture for others attempting similar deployments. It validates the uvm_disable_hmm=1 workaround as effective for this hardware-software combination. And it sets a performance expectation: the KVM VM's ~13 microsecond latency floor was a virtualization artifact, not a hardware limitation.

The Thinking Process

The assistant's thinking process is visible in several dimensions. The enthusiastic opening ("Excellent results!") signals relief and satisfaction after a long debugging session. The structured presentation of results — bullet points with bold formatting, explicit comparisons to the KVM baseline — shows analytical thinking: the assistant is not just reporting numbers but interpreting them in context. The phrase "This is a massive improvement" is a qualitative judgment that goes beyond the raw data, reflecting the assistant's understanding of how these numbers will impact the overall system performance.

The transition to "Let me now launch sglang" demonstrates forward-looking planning. The assistant does not dwell on the victory but immediately shifts focus to the next milestone. The parenthetical "First, let me check all the dependencies are in order" shows awareness of the many moving parts still remaining — the inference server has its own set of requirements (transformers version, FlashInfer, CUDA graphs, etc.) that could introduce new blockers.

The todo update is particularly revealing of the assistant's mental model. By marking P2P verification as completed and preparing for the deployment phase, the assistant is effectively closing a chapter and opening a new one. The structured task management reflects a methodical approach to a complex, multi-step engineering problem.

Conclusion

Message 569 is a pivotal moment in a long and challenging engineering effort. It represents the successful resolution of a fundamental infrastructure problem — enabling direct GPU-to-GPU communication in a virtualized environment — and the transition to the application layer. The P2P bandwidth numbers of 53 GB/s same-NUMA and 40 GB/s cross-NUMA validated the LXC container approach and the HMM fix, providing a solid foundation for the inference deployment that would follow. While assumptions about the sufficiency of this bandwidth and the stability of the fix remained to be tested, the message captures a genuine breakthrough: the moment when the infrastructure stopped being the bottleneck and the real work of model serving could begin.