From NaN to Virtualization: The Complete Arc of Deploying GLM-5-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs

Introduction

The deployment of a 744-billion-parameter Mixture-of-Experts (MoE) language model across eight cutting-edge Blackwell GPUs is never a straightforward affair. When that deployment takes place inside a virtualized Proxmox environment, with a novel 4-bit floating-point quantization format and a nightly build of the inference server, the path from "it crashes" to "it works" becomes a winding journey through kernel backends, autotuner configurations, memory tuning, and ultimately — a revelation about virtualization overhead that reframes everything.

This article synthesizes the complete arc of deploying GLM-5-NVFP4 on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, drawing from fourteen message-level analyses that together tell a story of systematic debugging, methodical benchmarking, and a pivotal insight that transformed the team's understanding of where the real bottleneck lay. The narrative moves through five distinct phases: crash resolution, baseline establishment, tuning exploration, hypothesis exhaustion, and infrastructure diagnosis. Each phase builds on the last, and together they form a case study in what it takes to get a bleeding-edge model running on bleeding-edge hardware in a non-ideal environment.

Phase 1: The NaN Crash and the NSA Backend Discovery

The journey began with a critical failure. Every time the server attempted decode — the autoregressive generation phase where the model produces one token at a time — it crashed with NaN (Not a Number) values. This was not a subtle performance issue; it was a hard blocker that made the model completely unusable [5].

The assistant's investigation revealed that the crash was specific to the SM120 architecture of the RTX PRO 6000 Blackwell GPUs. The SGLang inference server offers multiple attention backend options, and the default choices for the NSA (Non-Scalar Attention) mechanism were incompatible with SM120 hardware. Through systematic elimination, the assistant discovered that the --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags — forcing the attention computation to use TensorRT-LLM's implementation — produced coherent output where all other NSA backends failed [5][10].

This was the first major insight of the session: the FlashInfer NSA backends (flashmla_kv, flashmla_sparse) triggered device-side asserts on SM120 GPUs during decode, while the TRT-LLM path worked correctly. The fix was simple in retrospect — a single flag change — but reaching it required understanding the architecture-specific code paths in both FlashInfer and TensorRT-LLM, and the willingness to test each backend combination systematically [5][10].

Phase 2: Establishing the Baseline

With the NaN crash resolved, the assistant immediately pivoted to performance measurement. The first baseline benchmark, run with 64 concurrent requests at 256 input and 256 output tokens, showed approximately 205 output tokens per second and 472 total tokens per second [1]. This was the first reliable measurement of the model's performance on this hardware — a critical reference point for all subsequent tuning.

The assistant followed a deliberate methodology: verify correctness first (a simple "What is 3+5?" query returned "3 + 5 = 8" with proper reasoning), then measure performance. The benchmark used sglang.bench_serving with random data, infinite request rate, and carefully filtered output metrics. The concurrency of 34.89 (out of 64 requests) suggested the system was not fully saturated, and the 2.3:1 ratio of total tokens to output tokens indicated significant prefilling overhead [1].

A single-stream latency test — one request at a time — revealed a starkly different picture: only 11.10 output tokens per second over a 21.27-second duration [2]. This single-stream figure was dramatically lower than the batched throughput, revealing that the model's per-request latency was poor even though the system could aggregate throughput across many concurrent requests. This discrepancy between single-stream and batched performance would become a central mystery of the investigation [2][3].

Phase 3: Systematic Tuning and MoE Backend Exploration

With a working baseline established, the assistant turned to optimization. The first tuning parameter was --mem-fraction-static, increased from its default to 0.92 to maximize GPU memory utilization for the KV cache. CUDA graphs — a mechanism for pre-recording GPU operation sequences to reduce kernel launch overhead — were enabled and captured successfully without out-of-memory errors [4][7].

But the throughput remained stubbornly similar, in the 210–247 output tok/s range [4]. This was the first hint that the bottleneck was not where expected. If CUDA graphs — which directly address kernel launch overhead — produced no improvement, then kernel launch overhead was not the limiting factor.

The assistant then embarked on a systematic evaluation of MoE runner backends. SGLang offers three options: flashinfer_cutlass (using CUTLASS-generated fused MoE kernels), flashinfer_cutedsl (using NVIDIA's CuteDSL kernel authoring framework), and flashinfer_trtllm (using TensorRT-LLM-style kernels with autotuned tactic selection). The assistant discovered that flashinfer_trtllm was SM100-only and thus unavailable on the SM120 GPUs [4][9]. The remaining two backends were tested with identical benchmarks, and both produced comparable throughput — approximately 206 output tok/s for flashinfer_cutedsl versus 195–225 output tok/s for flashinfer_cutlass [2][4].

This was the second major insight: the MoE runner backend did not matter. The assistant had tested every available option and found them all within the same performance band. The bottleneck was not in the MoE kernel implementation [3].

Phase 4: The Moment of Synthesis

The assistant's conclusion in message 304 crystallized the investigation's state: "Same ~11 tok/s single-stream. The MoE runner backend doesn't change things — the bottleneck is elsewhere" [3]. This was a moment of intellectual closure — the recognition that a line of inquiry had been exhausted.

The assistant then returned to the best-known configuration (flashinfer_cutlass with CUDA graphs, trtllm NSA backends, --mem-fraction-static 0.92) and launched a comprehensive benchmark sweep [4][7]. The sweep tested concurrency levels from 1 to 64 with both balanced (256-in/256-out) and decode-heavy (128-in/512-out) workloads. The results confirmed the pattern: single-stream throughput at ~11 tok/s, scaling roughly linearly with concurrency to ~210–247 output tok/s at saturation [7][8].

The assistant also investigated expert parallelism (EP) at the user's suggestion, but concluded it was not beneficial. The model's 453 GB of MoE expert parameters far exceeded the 96 GB per GPU, making full replication impossible. Moreover, the small hidden dimension (3584) meant EP8 offered no communication advantage over tensor parallelism — the all-reduce volume was similar regardless [4][9][10].

A todo-list update in message 307 marked four high-priority tasks as completed: the NSA backend fix, baseline benchmarks, memory fraction and CUDA graph tuning, and the EP investigation [6][9]. The tone was one of closure — the assistant had explored the major optimization avenues and was ready to report results.

Phase 5: The Virtualization Revelation

Then the user asked a question that transformed everything: "Can this be cross-gpu latency? This is a VM in Proxmox, can we check if maybe something about that is at fault?" [11].

This question challenged the assistant's "compute-bound" conclusion with a specific, domain-informed alternative hypothesis. The user knew something the profiling tools hadn't captured: the system was running as a KVM/QEMU virtual machine under Proxmox, and virtualization can introduce overheads invisible to standard GPU monitoring [11][12].

The assistant's response was a masterclass in systematic diagnosis [12]. It formulated four specific mechanisms by which virtualization could affect cross-GPU communication: PCIe passthrough latency via IOMMU/VFIO overhead, NUMA misconfiguration, interrupt coalescing, and missing PCIe Access Control Services (ACS) overrides. Then it dispatched three parallel diagnostic commands targeting virtualization identity, NUMA topology, and GPU peer-to-peer capability [12].

The results were devastating. The nvidia-smi topo -p2p r command showed "NS" (Not Supported) for every GPU pair — meaning no direct GPU-to-GPU DMA was possible. Every cross-GPU transfer had to bounce through host memory via the hypervisor's emulated PCIe hierarchy. A bandwidth test confirmed the penalty: cross-GPU copies achieved 32.6 GB/s for large transfers, compared to 1033 GB/s for same-GPU copies — a 97% reduction [12][13].

But the critical finding was for small messages. At 12KB — roughly the size of a single all-reduce for one layer's hidden state — the cross-GPU bandwidth was only 0.90 GB/s with 13.7 µs latency [13]. For a model with 78 layers, each requiring all-reduce operations with small message sizes, this latency floor was catastrophic. The assistant summarized: "Without P2P, every NCCL all-reduce has to bounce through host RAM, adding latency and halving effective bandwidth" [13].

This was the moment when the investigation's frame shifted entirely. The question was no longer "which MoE backend is fastest?" or "how do we tune the server parameters?" but rather "how do we fix the virtualization bottleneck?" [13][14].

The Complete Picture: What Was Learned

The GLM-5-NVFP4 deployment on 8× RTX PRO 6000 Blackwell GPUs yielded several categories of knowledge, each valuable in its own right:

A validated configuration. The combination of --nsa-decode-backend trtllm, --nsa-prefill-backend trtllm, --moe-runner-backend flashinfer_cutlass, --attention-backend flashinfer, --fp8-gemm-backend cutlass, --mem-fraction-static 0.92, and --disable-custom-all-reduce produces a stable, working deployment on SM120 GPUs. This is non-trivial — many other combinations crash or produce NaN outputs [10].

Baseline performance metrics. Single-stream throughput of ~11 output tok/s, saturated throughput of ~210–247 output tok/s at 64 concurrent requests, with end-to-end latency of approximately 16–21 seconds depending on concurrency. The decode-heavy regime (128-in/512-out) shows slightly higher saturated throughput (~247 tok/s) than the balanced regime [7][8][10].

A compatibility matrix for SM120 GPUs. The flashinfer_trtllm MoE backend is SM100-only and unavailable. The flashmla_kv and flashmla_sparse NSA backends produce NaN on decode. The trtllm NSA backends work correctly. Both flashinfer_cutlass and flashinfer_cutedsl MoE runners work and perform similarly [10].

Negative results that save future effort. CUDA graphs capture successfully but provide no throughput improvement. Expert parallelism is not beneficial due to memory constraints and communication volume. The FlashInfer autotuner only applies to the TRT-LLM MoE path and has no configuration for SM120 GPUs [10].

The virtualization bottleneck diagnosis. The Proxmox/KVM environment lacks direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory. Small-message all-reduce operations — the dominant communication pattern in tensor-parallel MoE inference — suffer severe latency penalties (13.7 µs for 12KB messages) that accumulate across 78 layers per decode step [12][13].

The Broader Significance

This session is a case study in the challenges of deploying large language models at the frontier of hardware and software. The RTX PRO 6000 Blackwell GPUs represent NVIDIA's latest consumer/professional architecture (SM120), but the software ecosystem — FlashInfer, SGLang, NCCL — is still catching up. The NVFP4 quantization format is novel and requires specific kernel support. The Proxmox virtualization layer adds complexity that is easy to overlook but devastating in its performance impact.

The narrative arc of this session — from NaN crash to virtualization diagnosis — follows a pattern familiar to any engineer who has debugged complex systems: start with the obvious (the crash), establish a baseline (benchmarks), explore optimizations (tuning, backends), exhaust hypotheses (MoE runners, CUDA graphs, EP), and finally discover the root cause in an unexpected layer (virtualization). The user's question about Proxmox was the critical insight that connected the dots between the puzzling 55% power draw and the missing P2P support.

For practitioners deploying large models in virtualized environments, the lesson is clear: the virtualization layer is not transparent. Before tuning MoE backends or CUDA graphs, check nvidia-smi topo -p2p. Before concluding the system is compute-bound, measure cross-GPU latency at realistic message sizes. The bottleneck may not be where you think it is — and finding it requires both systematic methodology and the domain-specific intuition to ask the right questions.