The Virtualization Revelation: How a Proxmox VM Became the Hidden Bottleneck in Multi-GPU LLM Inference

Introduction

In the course of deploying a massive 744-billion-parameter Mixture-of-Experts language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message in an opencode coding session marked a dramatic pivot in understanding. After hours of methodically resolving software crashes, tuning server parameters, and benchmarking throughput, the assistant had produced a comprehensive summary concluding that the system was "compute-bound on small-batch MoE expert kernels." The numbers told a frustrating story: single-stream throughput languished at a mere 11 tokens per second, and even at full saturation with 64 concurrent requests, the system topped out at roughly 250 output tokens per second — far below the hardware's theoretical potential.

Then the user asked a simple question: "can this be cross-gpu latency? this is a vm in proxmox, can we check if maybe something about that is at fault?"

That question, and the assistant's response in message 314, transformed the entire investigation. What had been characterized as a compute-bound problem was suddenly revealed to be a virtualization-induced communication bottleneck. This article examines that pivotal message in detail: why it was written, what decisions it made, what assumptions it challenged, and how it reshaped the trajectory of the deployment effort.

The Context: A Deployment Nearing Its Limits

To understand message 314, we must first understand what preceded it. The session had been a long, grueling effort to deploy the GLM-5-NVFP4 model — a 744B-parameter MoE architecture with 256 experts, 78 layers, and NVFP4 quantization — on a remote machine running eight RTX PRO 6000 Blackwell GPUs. The assistant had already overcome numerous obstacles: resolving NaN crashes during decode by selecting the correct NSA attention backends (trtllm), tuning memory fractions and CUDA graphs, testing multiple MoE runner backends, and evaluating expert parallelism. Each of these efforts had been meticulously documented and benchmarked.

In message 311, the assistant presented a final summary of results. The tone was one of closure: "Here's the complete summary of where we stand." The analysis concluded that the GPUs were at 100% utilization but only 55% power draw, memory bandwidth was at 52-53%, and PCIe links were barely utilized. The diagnosis was that the system was "compute-bound on small-batch MoE expert kernels" — the GPU simply couldn't saturate its tensor cores on the small matrix operations required for single-token decode. The throughput scaled linearly with batch size, which seemed consistent with a compute-bound model.

But the user, reading this summary, noticed something that didn't add up. If the GPUs were truly compute-bound, why was power draw only 55% of the 600W TDP? Why was memory bandwidth only half utilized? And most importantly, the user knew something the assistant didn't: this machine was running inside a Proxmox hypervisor. The user's intuition — born of deep familiarity with the infrastructure — was that virtualization overhead might be the real culprit.

Message 314: The Diagnostic Pivot

The assistant's response in message 314 is a masterclass in data-driven diagnosis. It begins with a synthesis of five key findings extracted from the diagnostic commands just executed in the previous round (message 313):

Confirmed KVM/QEMU VM — The virtualization check returned kvm, and the DMI/BIOS data showed a QEMU i440FX machine type. This was not bare metal.
P2P is NS (Not Supported) — The nvidia-smi topo -p2p r command revealed that ALL GPU pairs showed "NS" for peer-to-peer read. This was the bombshell. It meant that no direct GPU-to-GPU DMA was possible — every cross-GPU transfer had to bounce through host memory via the CPU.
GPU NUMA affinity: N/A — The GPUs had no NUMA affinity exposed to the VM, meaning the Linux kernel couldn't optimize memory placement for GPU accesses.
Cross-GPU bandwidth: 32.6 GB/s — A simple P2P copy test showed 32.6 GB/s between GPU 0 and GPU 1, compared to 1033 GB/s for same-GPU copies. That's a 97% reduction in bandwidth when crossing GPUs.
No IOMMU messages in dmesg — The VM showed no IOMMU activity, consistent with VFIO passthrough without guest-visible IOMMU. The assistant then articulates the significance of finding #2: "The NS (Not Supported) P2P is likely the biggest issue. Without P2P, every NCCL all-reduce has to bounce through host RAM, adding latency and halving effective bandwidth." This is the critical insight — the assistant connects the virtualization finding to the specific communication pattern of the model. A 78-layer MoE model running with tensor parallelism 8 performs an all-reduce operation after every single layer. If each all-reduce is forced through host memory instead of direct GPU-to-GPU, the latency penalty accumulates across all 78 layers for every decode step.

The Decisions Made in This Message

Message 314 is not just an analysis — it's a decision point. The assistant makes several implicit and explicit decisions:

Decision 1: Accept the virtualization hypothesis. The assistant immediately pivots from the "compute-bound" conclusion of message 311 to a new hypothesis: virtualization-induced communication latency is the primary bottleneck. This is a significant intellectual shift. The earlier analysis had pointed to the small matrix sizes of NVFP4 expert kernels as the limiting factor. Now the assistant is willing to reconsider that conclusion in light of new evidence.

Decision 2: Design targeted experiments. Rather than stopping at the five findings, the assistant immediately launches two deeper investigations. The first is an NCCL all-reduce bandwidth test at multiple message sizes (12KB, 49KB, 196KB, 786KB) — sizes that correspond to the actual communication volumes in the model's all-reduce operations. The second is a PCIe topology dump from the VM's perspective. These are not random tests; they are carefully chosen to quantify the exact impact of the virtualization on the model's communication patterns.

Decision 3: Frame the problem in terms of latency, not bandwidth. This is the most sophisticated analytical move in the message. The assistant doesn't just note that cross-GPU bandwidth is lower — it implicitly recognizes that the latency of small transfers is the real issue. The all-reduce operations in a 78-layer MoE model involve many small messages (the hidden size of GLM-5 is 6144, so per-tensor all-reduce messages are on the order of tens of kilobytes). The bandwidth test is designed to measure latency at precisely these sizes.

Assumptions Made and Challenged

Message 314 operates on several assumptions, some explicit and some implicit:

Assumption 1: The P2P NS status is a VM configuration issue, not a hardware limitation. The assistant assumes that on bare metal, these GPUs would have P2P support. This is a reasonable assumption — the RTX PRO 6000 Blackwell GPUs are physically connected via PCIe Gen5 x16, and on a properly configured bare-metal system, P2P should be supported between GPUs on the same PCIe switch. The NS status is attributed to the VM's lack of IOMMU passthrough configuration.

Assumption 2: The small-message latency is the dominant factor. The assistant designs the NCCL test to measure latency at sizes from 12KB to 786KB, which corresponds to the hidden_size × dtype sizes used in the model's all-reduce operations. The assumption is that the model's performance is limited by the per-operation latency of these small all-reduces, not by aggregate bandwidth.

Assumption 3: The all-reduce pattern is the primary communication bottleneck. The assistant focuses on P2P copy bandwidth between GPU pairs, which simulates the all-reduce pattern used in tensor parallelism. This assumes that other communication patterns (e.g., the all-to-all used in expert parallelism) are not the dominant factor — a reasonable assumption given the earlier analysis that EP offered no advantage.

Assumption 4: The VM configuration is fixed and cannot be changed. The assistant doesn't ask about reconfiguring the Proxmox host to enable P2P passthrough. Instead, it focuses on understanding and quantifying the limitation. This is a pragmatic assumption — the assistant is working within the constraints of the environment as given.

The message also challenges a key assumption from the previous analysis: that the system was compute-bound. The earlier conclusion was based on GPU utilization at 100% and power at 55%, which the assistant interpreted as the GPU being busy but not fully utilized. The new evidence suggests that the GPU is spending a significant fraction of its time waiting for cross-GPU communication — the 100% utilization might reflect the GPU being occupied with small, inefficient kernels while stalled on communication, rather than being genuinely compute-bound.

Input Knowledge Required

To fully understand message 314, several pieces of prior knowledge are necessary:

Knowledge of NCCL and GPU P2P communication. The reader needs to understand what "P2P NS" means in the context of NVIDIA's NCCL library. Peer-to-peer communication allows GPUs to directly read each other's memory over the PCIe bus without involving the CPU. When P2P is not supported (NS), all cross-GPU transfers must go through system memory, which adds latency and consumes CPU and memory bandwidth.

Knowledge of tensor parallelism in MoE models. The significance of the P2P finding depends on understanding that GLM-5 uses tensor parallelism across 8 GPUs, which means every layer requires an all-reduce operation to synchronize the partial results from each GPU. With 78 layers, that's 78 all-reduce operations per decode step. If each all-reduce is slowed by virtualization overhead, the cumulative effect is devastating.

Knowledge of Proxmox/KVM virtualization. The reader needs to understand that Proxmox uses KVM (Kernel-based Virtual Machine) with QEMU for device emulation. GPU passthrough in KVM typically uses VFIO (Virtual Function I/O), which can support P2P DMA between passed-through devices — but only if the host is properly configured with IOMMU groups and ACS (Access Control Services) overrides.

Knowledge of the model's architecture. The message references hidden_size=6144 and the model's 78-layer structure. Understanding that each all-reduce operates on tensors of size proportional to hidden_size × number_of_GPUs is necessary to interpret the bandwidth test results.

Knowledge of the earlier benchmarking results. The message builds on the throughput numbers from message 311 — the 11 tok/s single-stream and ~250 tok/s saturated throughput. Without this context, the significance of the virtualization finding is diminished.

Output Knowledge Created

Message 314 produces several important pieces of knowledge that advance the investigation:

Quantified virtualization overhead. The NCCL bandwidth test produces concrete numbers: at 12KB (roughly the size of a single all-reduce for one layer's hidden state), the cross-GPU bandwidth is only 0.90 GB/s with 13.7 µs latency. At 49KB, it's 3.47 GB/s with 14.2 µs latency. These numbers quantify exactly how much overhead the VM adds to each all-reduce operation.

A new bottleneck hypothesis. The message establishes a clear, testable hypothesis: the primary bottleneck is virtualization-induced latency on small cross-GPU transfers, not compute-bound kernel execution. This hypothesis can be tested by comparing against bare-metal performance or by trying to mitigate the virtualization overhead.

A diagnostic framework. The message establishes a methodology for diagnosing multi-GPU communication bottlenecks: check virtualization status, verify P2P support, measure bandwidth at realistic message sizes, and examine PCIe topology. This framework is reusable for any multi-GPU deployment.

A priority shift. The message implicitly reprioritizes the next steps. Earlier, the focus was on tuning server parameters and trying different backends. Now, the focus shifts to addressing the virtualization issue — either by reconfiguring the host, moving to bare metal, or finding software workarounds.

The Thinking Process Visible in the Message

The assistant's reasoning in message 314 follows a clear arc:

Step 1: Synthesize the diagnostic data. The assistant begins by listing five findings from the commands executed in the previous round. This is not just a data dump — each finding is selected for its relevance to the bottleneck question. The findings are ordered from general (virtualization confirmed) to specific (P2P NS, NUMA N/A, bandwidth numbers, IOMMU absence).

Step 2: Identify the critical finding. The assistant singles out P2P NS as "likely the biggest issue" and explains why: without P2P, all-reduce operations bounce through host memory, adding latency. The comparison to bare-metal PCIe Gen5 x16 performance (~64 GB/s) provides a reference point for how much performance is being lost.

Step 3: Design targeted experiments. Rather than speculating, the assistant immediately launches two experiments. The NCCL bandwidth test is particularly well-designed — it tests multiple message sizes that correspond to the actual communication volumes in the model's all-reduce operations. The PCIe topology check provides additional context about how the GPUs are connected from the VM's perspective.

Step 4: Let the data speak. The message ends with the raw output of the two experiments. The assistant doesn't provide a lengthy interpretation — the numbers speak for themselves. The bandwidth test shows a clear pattern: at small message sizes (12KB, typical of per-layer all-reduce), bandwidth plummets to 0.90 GB/s with 13.7 µs latency. The PCIe topology shows a flat i440FX bus with no direct GPU-to-GPU connections visible to the VM.

The thinking is analytical and data-driven throughout. The assistant doesn't jump to conclusions — it collects evidence, identifies patterns, and designs experiments to test hypotheses. The message is a model of systematic debugging.

Mistakes and Incorrect Assumptions

While message 314 is largely correct in its analysis, there are some limitations and potential oversights:

The assumption that P2P is the only issue. The message focuses heavily on P2P NS as the bottleneck, but the NUMA affinity issue (finding #3) could also be significant. If the GPUs are on different NUMA nodes from the vCPUs, every GPU memory access from the CPU side (including the bounce buffer operations for P2P) would cross NUMA boundaries, adding further latency. The message doesn't explore this angle.

The bandwidth test may not fully represent NCCL all-reduce. The test uses simple copy_ operations between GPU buffers, which simulates P2P write. NCCL all-reduce uses more complex algorithms (ring, tree, etc.) that may have different performance characteristics. The test provides a lower bound on latency but may not capture the full overhead of NCCL's synchronization and reduction operations.

No comparison to intra-GPU all-reduce. The message doesn't measure the latency of an actual NCCL all-reduce operation within a single GPU (which would be essentially zero) or between GPUs on the same PCIe switch in an ideal configuration. Having these baselines would strengthen the conclusion.

The PCIe topology is from the VM's perspective. The lspci -tv output shows the emulated PCIe topology, which may not reflect the physical topology. The GPUs might be on different physical PCIe switches or root complexes, which would affect P2P performance even on bare metal.

Despite these limitations, the message's core insight is sound: virtualization overhead on cross-GPU communication is a significant factor in the observed throughput limitations. The message successfully pivots the investigation from a compute-bound hypothesis to a communication-bound one, setting the stage for the next phase of work.

Conclusion

Message 314 is a turning point in the deployment of GLM-5-NVFP4 across eight Blackwell GPUs. What began as a story about software crashes, backend selection, and kernel tuning suddenly became a story about virtualization overhead, PCIe passthrough, and the hidden costs of running GPU workloads inside a hypervisor.

The message's power lies not in any single discovery but in the synthesis of multiple data points into a coherent new hypothesis. The assistant takes five separate findings — KVM detection, P2P NS status, missing NUMA affinity, reduced cross-GPU bandwidth, and absent IOMMU — and weaves them into a narrative that explains the frustrating throughput ceiling. The subsequent bandwidth test at realistic message sizes provides the quantitative evidence needed to support this narrative.

For the broader investigation, this message represents a fundamental reframing. The question is no longer "which MoE backend is fastest?" or "how do we tune the server parameters?" but rather "how do we fix the virtualization bottleneck?" This shift in framing will determine the next steps: investigating NCCL tuning options for VM environments, exploring bare-metal deployment, or reconfiguring the Proxmox host to enable P2P passthrough.

In the end, message 314 is a reminder that in complex systems, the bottleneck is often not where you think it is. The assistant had spent hours tuning software parameters and testing different backends, all while the real bottleneck — a missing configuration in the hypervisor — lay invisible beneath the application layer. It took the user's intuition about the Proxmox environment to point the investigation in the right direction. And it took the assistant's systematic diagnostic approach to turn that intuition into quantifiable evidence.