Diagnosing Virtualization-Induced Bottlenecks in Multi-GPU Inference: A Proxmox Investigation

Introduction

In the high-stakes world of large language model deployment, performance debugging often requires tracing a thread from the application layer all the way down through the operating system, the hypervisor, and ultimately to the bare-metal hardware. This article examines a pivotal moment in a coding session where an AI assistant and a user collaborate to deploy the GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts (MoE) architecture—across eight NVIDIA RTX PRO 6000 Blackwell GPUs. After weeks of environment setup, driver installation, flash-attn compilation heroics, and iterative debugging of NaN crashes during inference, the team had achieved a working deployment. Yet a puzzling performance profile remained: the GPUs reported 100% utilization but drew only 55% of their rated power, and single-stream throughput languished at roughly 11 tokens per second. The user, suspecting that the virtualization layer might be the culprit, asked a pointed question: "Can this be cross-GPU latency? This is a VM in Proxmox, can we check if maybe something about that is at fault?"

The message that follows—message index 313 in the conversation—is the assistant's response to that question. It is a masterclass in systematic diagnostic reasoning, combining deep knowledge of virtualization internals, GPU interconnect architecture, and NCCL communication patterns. In a single round, the assistant formulates four specific hypotheses, dispatches three parallel diagnostic shell commands, and begins to uncover evidence that the Proxmox hypervisor is indeed imposing a critical performance penalty on cross-GPU communication. This article dissects that message in detail, examining the reasoning that drove it, the assumptions it made, the knowledge it required, and the new understanding it produced.

The Context: A Performance Puzzle

To appreciate the significance of message 313, one must understand the journey that led to it. The team had been working for days to deploy GLM-5-NVFP4 on a machine running Ubuntu 24.04 with eight RTX PRO 6000 GPUs. This is a Blackwell-architecture GPU (compute capability SM120) with 96 GB of HBM3e memory per card—a formidable but still constrained platform for a 744B MoE model that requires roughly 453 GB just for its expert weights. The model uses NVFP4 quantization (4-bit floating point) and employs a Mixture-of-Experts architecture with 256 experts, of which 8 are activated per token.

After resolving a critical NaN crash during decode—which required selecting the TRT-LLM backend for the model's NSA (Native Sparse Attention) mechanism—the team had established a working configuration. The assistant ran comprehensive benchmarks across varying concurrency levels, producing a clear picture: single-stream throughput of ~11 tok/s, scaling roughly linearly with batch size up to ~210 tok/s at concurrency 64. The GPUs showed 100% utilization in nvidia-smi, but power draw hovered at only 55% of the 600W TDP (approximately 330W). Memory bandwidth utilization was also modest at 52-53%.

These numbers told a story of a system that was compute-bound on small kernels rather than memory-bound or PCIe-bandwidth-bound. The assistant had already explored and ruled out several potential optimizations: CUDA graphs (captured successfully but didn't help), expert parallelism (impossible due to memory constraints), alternative MoE runner backends (all performed similarly), and FlashInfer autotuning (only applies to a different kernel path). The bottleneck seemed to be the sheer number of sequential operations per decode step—78 layers, each with MLA attention, MoE expert dispatch, and all-reduce synchronization—but the exact why remained elusive.

This is where the user's question about Proxmox virtualization entered the picture. It was a hypothesis that could explain the paradoxical 100% utilization / 55% power signature: if the GPUs were spending a significant fraction of their time waiting on cross-GPU communication that was being slowed by the hypervisor, they would appear busy (utilization counts any active cycle) but wouldn't be drawing full power (which requires sustained compute-unit activity).

The Assistant's Reasoning: Four Hypotheses

The assistant's response begins with an explicit reasoning section that demonstrates both deep system knowledge and a structured approach to diagnosis. Rather than jumping straight to running commands, the assistant first articulates why a Proxmox VM could explain the observed symptoms, listing four specific mechanisms:

1. PCIe passthrough latency (IOMMU/VFIO overhead): When GPUs are passed through to a VM via VFIO (Virtual Function I/O), every memory-mapped I/O (MMIO) access must traverse the hypervisor's IOMMU translation tables. For NCCL all-reduce operations—which involve many small inter-GPU messages—this per-access overhead can dominate. The assistant correctly identifies that cross-GPU NCCL communication in a VM environment goes through the hypervisor's IOMMU translation layer, adding latency to every DMA operation.

2. NUMA misconfiguration: If the VM's virtual CPUs are not aligned with the physical NUMA node that the GPUs are attached to, every GPU memory access could cross NUMA boundaries. This is a common pitfall in VM deployments where the hypervisor may not expose NUMA topology correctly to the guest, or where the VM's vCPU pinning is misconfigured. A NUMA-misaligned system can see dramatically higher latency for GPU memory operations.

3. Interrupt coalescing and vIOMMU overhead: Proxmox (which uses KVM under the hood) may add latency to GPU interrupts and DMA completion signals. Interrupt coalescing—where the hypervisor batches multiple interrupts before delivering them to the guest—can reduce throughput for latency-sensitive operations while improving overall host efficiency. For GPU kernels that synchronize frequently, this added latency can be crippling.

4. Missing ACS (Access Control Services) override: This is perhaps the most technically subtle point. PCIe Access Control Services (ACS) can prevent direct peer-to-peer (P2P) transfers between devices behind a PCIe switch. In a bare-metal Linux system, the kernel can override ACS to enable P2P. But in a VM, the guest kernel may not have the ability to perform this override, forcing all GPU-to-GPU traffic to go through the root complex (and potentially through host memory) rather than using direct P2P transfers. This turns every all-reduce into a multi-hop journey.

These four hypotheses are not random guesses—they are grounded in the specific architecture of KVM-based virtualization (which Proxmox uses) and the known behavior of NVIDIA GPUs in virtualized environments. The assistant demonstrates a sophisticated understanding that performance problems in ML workloads often have root causes far removed from the model code itself.

The Investigation: Three Parallel Diagnostic Commands

After laying out the hypotheses, the assistant dispatches three bash commands via SSH to the remote machine. These are executed in parallel (as is the convention in this coding session's tool-calling protocol), each targeting a different layer of the system's configuration. Let us examine each command and its output in detail.

Command 1: Virtualization Identity and IOMMU Status

The first command performs a basic identity check:

ssh 10.1.230.175 'echo "=== Virtualization check ===" && systemd-detect-virt 2>/dev/null || echo "no systemd-detect-virt" && echo && echo "=== DMI/BIOS ===" && cat /sys/class/dmi/id/product_name 2>/dev/null && cat /sys/class/dmi/id/sys_vendor 2>/dev/null && echo && echo "=== IOMMU ===" && dmesg 2>/dev/null | grep -i "iommu\|vfio\|passthrough\|dmar" | head -20 && echo && echo "=== CPU model ===" && grep "model name" /proc/cpuinfo | head -1'

The output confirms three critical facts:

Command 2: NUMA Topology and GPU Affinity

The second command probes NUMA configuration and GPU placement:

ssh 10.1.230.175 'echo "=== NUMA topology ===" && numactl --hardware 2>/dev/null || echo "numactl not available" && echo && echo "=== GPU NUMA affinity ===" && for i in $(seq 0 7); do echo -n "GPU$i numa_node: "; cat /sys/bus/pci/devices/$(nvidia-smi -i $i --query-gpu=pci.bus_id --format=csv,noheader | tr ":" "/" | sed "s/^/0000/")/numa_node 2>/dev/null || echo "N/A"; done && echo && echo "=== PCIe ACS ===" && lspci -vvs 00:10.0 2>&1 | grep -i "acs\|access control" | head -5'

The output reveals:

Command 3: Peer-to-Peer Capability and Bandwidth Test

The third and most revealing command checks GPU P2P connectivity and runs a bandwidth benchmark:

ssh 10.1.230.175 'echo "=== P2P check ===" && nvidia-smi topo -p2p r 2>/dev/null | head -15 && echo && echo "=== NCCL debug test ===" && source ~/ml-env/bin/activate && NCCL_DEBUG=INFO python3 -c "
import torch
import torch.distributed as dist
# Quick P2P bandwidth test between GPU 0 and 1
a = torch.randn(1024, 1024, device=\"cuda:0\")
b = torch.empty(1024, 1024, device=\"cuda:1\")
torch.cuda.synchronize()
import time
start = time.time()
for i in range(100):
    b.copy_(a)
torch.cuda.synchronize()
elapsed = time.time() - start
bw = 100 * 4 * 1024 * 1024 / elapsed / 1e9
print(f\"GPU0->GPU1 copy bandwidth: {bw:.2f} GB/s ({elapsed*10:.1f} ms per copy)\")
# Same GPU baseline
c = torch.randn(1024, 1024, device=\"cuda:0\")
d = torch.empty(1024, 1024, device=\"cuda:0\")
torch.cuda.synchronize()
start = time.time()
for i in range(100):
    d.copy_(c)
torch.cuda.synchronize()
elapsed2 = time.time() - start
bw2 = 100 * 4 * 1024 * 1024 / elapsed2 / 1e9
print(f\"GPU0->GPU0 copy bandwidth: {bw2:.2f} GB/s ({elapsed2*10:.1f} ms per copy)\")
" 2>&1 | grep -v "^$"'

The output is devastating:

P2P Status: All "NS" (Not Supported)

The nvidia-smi topo -p2p r command shows that every GPU pair reports "NS" in the P2P matrix. The legend confirms: "X = Self, OK = Status Ok, CNS = Chipset not supported." There are no "OK" entries anywhere. This means that no direct peer-to-peer communication path exists between any two GPUs in this VM. Every inter-GPU transfer must go through the host's root complex—and in a VM, that means through the hypervisor's emulated PCIe hierarchy.

This is the smoking gun. In a bare-metal system with eight GPUs on the same PCIe switch, one would expect to see "OK" entries between GPUs that share a switch. The "NS" status across the board indicates that either:

Assumptions and Their Validity

The assistant's investigation rests on several assumptions, most of which are validated by the evidence:

Assumption 1: Virtualization could explain the utilization/power paradox. This is supported by the P2P "NS" finding. If GPUs cannot communicate directly, NCCL must route all-reduce through host memory, adding latency that keeps GPUs in a "busy-waiting" state. The GPU appears utilized (it's actively polling or waiting on synchronization) but isn't doing productive compute work (hence lower power draw).

Assumption 2: The diagnostic commands would reveal actionable information. This proved correct. The P2P check immediately identified a critical limitation, and the bandwidth test quantified the cross-GPU performance.

Assumption 3: The user's Proxmox environment uses standard KVM/QEMU. Confirmed by systemd-detect-virt returning "kvm" and the BIOS string identifying QEMU.

Assumption 4: The bandwidth test using torch.Tensor.copy_ is representative of NCCL communication patterns. This is partially valid. The copy_ operation is a simple point-to-point transfer, while NCCL all-reduce involves more complex patterns (scatter-reduce-gather or tree-based algorithms). However, if even a simple P2P copy shows 30× degradation versus same-GPU, the all-reduce overhead will be at least as severe.

Potential oversight: No small-message latency test. The assistant's bandwidth test uses 4 MB messages (1024×1024 floats). NCCL all-reduce in transformer inference often involves much smaller messages—on the order of a few kilobytes for attention activations or a few megabytes for MLP intermediates. The assistant does not test with small message sizes, which would likely show even worse relative performance. The 32.59 GB/s figure is impressive for large transfers but may not reflect the latency-dominated regime of actual inference communication.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

GPU architecture: Understanding that RTX PRO 6000 uses the Blackwell architecture (SM120 compute capability), that NVFP4 is a 4-bit floating-point quantization format, and that MoE models have unique communication patterns due to expert routing.

Virtualization internals: Knowledge of KVM/QEMU, VFIO passthrough, IOMMU, PCIe ACS, and how these components interact to affect GPU performance. The concept of "NS" in nvidia-smi topo -p2p output and its implications for NCCL communication.

NCCL and distributed inference: Understanding tensor parallelism (TP), all-reduce operations, and how communication overhead scales with model size and batch size. The difference between bandwidth-bound and latency-bound regimes in collective communication.

Performance analysis: The ability to interpret the utilization/power paradox—100% GPU utilization with only 55% power draw suggests the GPU is active but not doing sustained compute work. This pattern is characteristic of latency-bound workloads where the GPU spends significant time waiting for data.

Linux system administration: Familiarity with systemd-detect-virt, numactl, nvidia-smi topo, lspci, and dmesg for system diagnostics.

Output Knowledge Created

This message produces several concrete findings that advance the investigation:

  1. Confirmed virtualization environment: The system is definitively identified as a KVM/QEMU virtual machine (Proxmox), ruling out bare-metal or alternative hypervisor configurations.
  2. No GPU peer-to-peer support: The nvidia-smi topo -p2p output shows "NS" for all GPU pairs, meaning no direct GPU-to-GPU communication path exists. This is a fundamental limitation imposed by the virtualization layer.
  3. Cross-GPU bandwidth quantified: Large-message P2P copy achieves ~32.59 GB/s, which is reasonable for PCIe but ~30× slower than same-GPU copies (~1033 GB/s). This establishes a baseline for understanding communication overhead.
  4. NUMA topology documented: Two NUMA nodes with 60 CPUs each, providing the foundation for further NUMA-aware tuning if needed.
  5. IOMMU status unclear: The absence of IOMMU messages in dmesg is noted but not yet resolved, leaving a potential avenue for further investigation.
  6. Hypothesis validated: The user's intuition that virtualization could be the bottleneck is strongly supported by the evidence. The assistant's response not only confirms the suspicion but provides specific mechanisms (missing P2P, IOMMU overhead, potential NUMA misalignment) that explain the observed performance.

The Thinking Process: A Window into Diagnostic Reasoning

What makes this message particularly valuable is the visible reasoning structure. The assistant does not simply run commands and report output—it first articulates why each hypothesis is plausible, creating a mental model that connects symptoms to root causes. This is diagnostic reasoning at its best:

  1. Observe the anomaly: 100% GPU utilization + 55% power draw + low throughput = something is keeping GPUs busy without doing compute work.
  2. Generate hypotheses based on system architecture: The assistant knows that Proxmox uses KVM, that GPU passthrough involves IOMMU, that PCIe ACS can block P2P, and that NUMA misconfiguration is common in VMs. Each hypothesis is grounded in a known virtualization behavior.
  3. Design tests to discriminate between hypotheses: The three commands are carefully chosen to test different layers. Command 1 confirms virtualization type and IOMMU status. Command 2 checks NUMA configuration. Command 3 directly tests the most critical hypothesis (P2P capability) and quantifies the penalty.
  4. Interpret results in context: The "NS" P2P output is immediately recognized as significant. The bandwidth numbers are compared to theoretical PCIe limits and to same-GPU baselines. The assistant does not over-interpret the 32.59 GB/s figure but instead notes that the relevant regime is small messages, where latency dominates.
  5. Identify next steps implicitly: By documenting what was found and what remains unclear (the truncated NUMA affinity, the missing IOMMU messages), the assistant sets the stage for deeper investigation in subsequent messages. This structured approach is what separates effective debugging from random experimentation. The assistant treats the system as a set of interacting layers (application → CUDA → NCCL → PCIe → IOMMU → hypervisor → hardware) and systematically tests each layer's contribution to the observed problem.

Conclusion

Message 313 represents a turning point in the GLM-5-NVFP4 deployment effort. Before this message, the team had a working but puzzlingly slow system—GPUs at full utilization but half power, throughput that scaled linearly with batch size but never reached the GPU's potential. The user's question about Proxmox virtualization opened a new line of investigation, and the assistant's response delivered a masterful diagnostic execution.

The key findings—no GPU peer-to-peer support, cross-GPU bandwidth 30× slower than same-GPU, and a confirmed KVM/QEMU environment—provide the missing piece of the puzzle. The bottleneck is not in the model architecture, the quantization format, the MoE runner backend, or the CUDA graph capture. It is in the virtualization layer itself, which prevents direct GPU-to-GPU communication and forces all cross-GPU traffic through the hypervisor's emulated PCIe hierarchy.

For practitioners deploying large models in virtualized environments, this message offers a cautionary tale and a diagnostic template. The combination of nvidia-smi topo -p2p for P2P status, a simple PyTorch bandwidth test for quantifying the penalty, and NUMA topology inspection for detecting misconfiguration should be part of any pre-deployment checklist. The lesson is clear: when deploying multi-GPU inference workloads in VMs, the virtualization layer is not a transparent abstraction—it is an active participant in the system's performance characteristics, and its effects must be measured and understood.

The assistant's response also demonstrates the value of structured diagnostic reasoning in complex systems. By articulating hypotheses before running tests, the assistant transforms what could have been a scattershot collection of commands into a coherent investigation. Each command has a purpose, each result has an interpretation, and the overall narrative advances the team's understanding of their system. This is the essence of effective collaboration between human intuition and AI-assisted analysis—the user provides the domain-specific insight ("could this be the VM?"), and the assistant provides the systematic execution and technical depth to validate or refute the hypothesis.