Deploying GLM-5-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs: From NaN Crashes to Virtualization Bottlenecks

Introduction

The deployment of a 744-billion-parameter Mixture-of-Experts (MoE) language model across eight cutting-edge Blackwell GPUs inside a virtualized Proxmox environment represents a convergence of challenges that test the limits of systematic debugging, performance engineering, and infrastructure analysis. When a team set out to deploy GLM-5-NVFP4 — a model using DeepSeek Sparse Attention (DSA), NVFP4 quantization, and 256 experts across 75 MoE layers — on eight NVIDIA RTX PRO 6000 GPUs, they encountered a cascade of obstacles that spanned the entire stack: from CUDA kernel numerical stability to autotuner configuration gaps to virtualization-induced PCIe latency.

This article synthesizes the work captured in Segment 2 of an opencode coding session, tracing the journey from a catastrophic NaN crash during decode, through baseline benchmarking and server tuning, to the ultimate identification of virtualization-induced PCIe latency as the primary performance bottleneck. The narrative moves through five distinct phases — crash resolution, baseline establishment, tuning exploration, hypothesis exhaustion, and infrastructure diagnosis — each building on the last to form a case study in what it takes to get a bleeding-edge model running on bleeding-edge hardware in a non-ideal environment.

The themes that emerge — systematic hypothesis testing, the gap between intuition and measurement, the importance of understanding hardware topology, and the value of methodical tooling — are broadly applicable to anyone working at the intersection of large models and non-standard GPU infrastructure.

Part I: The NaN Crisis and the NSA Backend Breakthrough

The session's central blocker was a catastrophic failure mode: every time the server attempted decode — the autoregressive generation phase where the model produces one token at a time — it crashed with a CUDA device-side assert: "probability tensor contains either inf, nan or element < 0." This NaN (Not-a-Number) error during decode was not a random glitch but a systematic failure that persisted across multiple configuration attempts.

The root cause lay in the interaction between the model's DSA attention mechanism and the SM120 GPU architecture of the RTX PRO 6000. GLM-5-NVFP4 uses DeepSeek Sparse Attention, which forces SGLang to use specialized NSA (Native Sparse Attention) backends. The available backends — flashmla_kv, flashmla_sparse, and trtllm — implement the same mathematical operation but through different CUDA kernel paths. On the SM120 architecture (which has only 101KB of shared memory compared to 228KB on datacenter SM100 Blackwell), the flashmla_kv and flashmla_sparse backends silently produced numerical instability.

The breakthrough came when the assistant launched the server with --nsa-decode-backend trtllm --nsa-prefill-backend trtllm and received the first coherent output. The trtllm backend, backed by NVIDIA's TensorRT-LLM library, handled the SM120 architecture correctly where the flashmla-based implementations failed. This was confirmed through progressively more demanding tests: a simple "What is 2+2?" query, a longer generation with 256 tokens, and a code-generation task requesting a Python is_prime function — all produced correct, well-formatted output.

The debugging methodology that led to this fix is worth examining. The assistant maintained a structured table of configuration attempts, each isolating one or two variables. This systematic elimination of hypotheses — ruling out DeepGemm, CUDA graphs, KV cache dtype, and attention backends in turn — is textbook scientific debugging applied to ML infrastructure. The key insight was recognizing that the model card's recommended configuration, designed for datacenter SM100 Blackwell, was not applicable to the SM120 workstation variant. This pattern — where default configurations optimized for datacenter hardware silently fail on workstation-class GPUs — is becoming increasingly common as the gap between datacenter and consumer/professional GPU architectures widens.

Part II: Establishing Baseline Throughput and Overcoming Tooling Friction

With the NaN crash resolved, the assistant pivoted to performance measurement. This transition from "does it work?" to "how fast does it work?" proved surprisingly challenging, as the standard benchmarking tools were not designed for reasoning models.

The first benchmark attempt crashed with a 401 Unauthorized error because the bench_serving tool conflated the served model name (glm-5) with the HuggingFace repository identifier (lukealonso/GLM-5-NVFP4). The second attempt corrected this with a --tokenizer flag but then crashed during metrics calculation because the reasoning model returns content: null during the thinking phase, and the tokenizer could not handle a null generated_text. The third attempt switched to the native SGLang backend, which worked but produced broken per-token metrics because the non-streaming /generate endpoint cannot provide token-level timing.

The assistant's response to these failures is instructive. Rather than trying to fix the benchmarking tool — a major engineering detour that would have consumed hours — it adapted its measurement strategy to work within the tool's limitations. It identified aggregate output token throughput as the one reliable metric and used that as its north star. The first meaningful baseline — 144 output tok/s and 304 total tok/s with 32 concurrent requests — was established with the assistant explicitly noting that "All 32 requests succeeded — no crashes under concurrent load."

Subsequent benchmarks with 64 concurrent requests pushed throughput to approximately 225 output tok/s and 516 total tok/s. This became the reference point against which all tuning efforts would be measured. The concurrency of approximately 35 (out of 64 requests) suggested the system was not fully saturated, and the roughly 2.3:1 ratio of total tokens to output tokens indicated significant prefilling overhead.

A single-stream latency test — one request at a time — revealed a starkly different picture: only 11.10 output tokens per second over a 21.27-second duration. This single-stream figure was dramatically lower than the batched throughput, revealing that the model's per-request latency was poor even though the system could aggregate throughput across many concurrent requests. This discrepancy between single-stream and batched performance would become a central mystery of the investigation.## Part III: Server Tuning and the PCIe Bottleneck Revelation

With a baseline established, the assistant systematically explored server tuning parameters. Increasing --mem-fraction-static from 0.85 to 0.92 expanded the KV cache from 370K to 498K tokens without triggering out-of-memory errors. Enabling CUDA graphs — a GPU optimization that pre-compiles kernel launch sequences to reduce driver overhead — captured successfully for batch sizes up to 64.

But the results were disappointing. Despite successful CUDA graph capture, throughput remained stubbornly in the 210–247 tok/s range — a mere 5% improvement over the baseline. This negative result was itself diagnostic: if CUDA graphs (which specifically optimize kernel launch overhead) did not help, then the bottleneck must be in the computation or communication that happens between kernel launches, not in the launches themselves.

The assistant articulated the conclusion clearly: "The throughput is capped around 200-236 tok/s regardless of CUDA graphs. This suggests the bottleneck is the MoE expert computation + all-reduce over PCIe, not kernel launch overhead." This marked a critical shift in mental model — from a "kernel launch overhead" frame to a "communication-bound" frame.

The single-stream latency test confirmed the severity of the bottleneck: approximately 11 tokens per second for a single request. With 61 MoE layers and 8 GPUs performing all-reduce after each layer, the PCIe interconnect was being hammered by small messages that could not saturate the theoretical bandwidth but were latency-limited by the round-trip time through the host bridge. Each all-reduce operation requires synchronizing hidden states across all eight GPUs, and with 61 layers, this synchronization happens 61 times per generated token. The cumulative latency of these small-message synchronizations — each taking microseconds but adding up across layers and tokens — created a performance ceiling that no amount of kernel tuning could突破.

Part IV: Expert Parallelism — When Intuition Meets Arithmetic

The user, recognizing the PCIe bottleneck, posed a provocative question: "Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?" This launched a deep investigation into Expert Parallelism (EP) as an alternative to Tensor Parallelism (TP).

The initial analysis was promising. EP replaces TP's all-reduce (which requires every GPU to synchronize after every layer) with all-to-all communication (which only requires communication for token routing). For MoE models with large hidden dimensions, this can dramatically reduce cross-GPU traffic. The user went further, proposing an even more elegant idea: replicate all 256 experts on every GPU, eliminating cross-GPU communication for expert computation entirely. Each GPU would independently compute its assigned expert activations using locally stored weights, with only the initial dispatch and final collection requiring communication.

The arithmetic, however, was unforgiving. The assistant wrote a comprehensive Python script that computed the exact memory footprint of every model component. The results were stark:

• MoE experts (NVFP4): 453.0 GB across 75 layers
• Attention (BF16): 25.7 GB across all 78 layers
• Shared experts (BF16): 5.7 GB
• Dense MLPs (BF16): 1.36 GB
• Embeddings (BF16): 1.90 GB
• Total: approximately 487.7 GB With TP8, each GPU holds about 61 GB — comfortably within the 96 GB limit. But full replication would require 453 GB of experts on every GPU, which is 4.7× the available memory. The verdict was unambiguous: "Fits? NO." The analysis then turned to standard EP8 (distributing experts across GPUs without replication). Here, the assistant discovered a counterintuitive result: the communication volume per token was nearly identical between TP8 and EP8 — approximately 21 KB for TP all-reduce versus 24 KB for EP all-to-all. The model's hidden size of 6144 is relatively small by modern standards, meaning the all-reduce payload is modest regardless of parallelism strategy. A detailed computational model confirmed the finding: both TP8 and EP8 exhibited a roughly 4.8× communication-to-compute ratio. For every microsecond spent computing, nearly five microseconds were spent waiting for data to traverse the PCIe bus. The theoretical advantage of EP — embarrassingly parallel expert computation — was neutralized by the reality that both approaches required similar amounts of cross-GPU data movement for this particular model. The small hidden dimension meant that the all-reduce in TP was already cheap, and the all-to-all in EP was not significantly cheaper. This finding is a powerful reminder that parallelism strategies cannot be evaluated in isolation. The optimal strategy depends on the specific model parameters — hidden dimension, number of experts, intermediate expansion factor, quantization format — and the specific hardware topology. What works for a 405B dense model with a large hidden dimension may not work for a 744B MoE model with aggressive quantization and a small hidden dimension.## Part V: The Virtualization Revelation The investigation took another turn when the assistant began measuring actual PCIe throughput. The results were shocking: large transfers achieved approximately 32 GB/s (close to the theoretical PCIe Gen5 x16 limit), but small messages — the kind used in all-reduce operations — achieved only about 1 GB/s. This discrepancy pointed to a deeper issue. The assistant checked the GPU topology with nvidia-smi topo -m and discovered that all eight GPUs were connected via PHB (PCIe Host Bridge) links, with no direct GPU peer-to-peer (P2P) support. The P2P Status showed "NS" (Not Supported) for all GPU pairs. Further investigation confirmed that the system was a KVM/QEMU virtual machine running under Proxmox. In such an environment, the hypervisor does not expose direct GPU P2P DMA capabilities to the guest VM. All cross-GPU transfers must traverse the host memory — going from GPU → host memory via PCIe, then from host memory → destination GPU via another PCIe hop. This doubles the PCIe traversal for every cross-GPU communication and, more importantly, introduces the latency of two host memory copies. The bandwidth test results told the story: ~32 GB/s for large transfers (which can amortize the per-transfer overhead) but only ~1 GB/s for 4KB messages typical of all-reduce (where the per-transfer latency dominates). The virtualization layer was adding significant latency to every small cross-GPU communication, and with 61 MoE layers each requiring an all-reduce, this latency multiplied into a severe throughput bottleneck. The user's question — "Can this be cross-gpu latency? This is a VM in Proxmox, can we check if maybe something about that is at fault?" — was the critical insight that connected the dots. The assistant's response was a masterclass in systematic diagnosis. It formulated four specific mechanisms by which virtualization could affect cross-GPU communication: PCIe passthrough latency via IOMMU/VFIO overhead, NUMA misconfiguration, interrupt coalescing, and missing PCIe Access Control Services (ACS) overrides. Then it dispatched three parallel diagnostic commands targeting virtualization identity, NUMA topology, and GPU peer-to-peer capability. The results were devastating. The nvidia-smi topo -p2p r command showed "NS" (Not Supported) for every GPU pair — meaning no direct GPU-to-GPU DMA was possible. Every cross-GPU transfer had to bounce through host memory via the hypervisor's emulated PCIe hierarchy. A bandwidth test confirmed the penalty: cross-GPU copies achieved 32.6 GB/s for large transfers, compared to 1033 GB/s for same-GPU copies — a 97% reduction. But the critical finding was for small messages. At 12KB — roughly the size of a single all-reduce for one layer's hidden state — the cross-GPU bandwidth was only 0.90 GB/s with 13.7 µs latency. For a model with 78 layers, each requiring all-reduce operations with small message sizes, this latency floor was catastrophic. The assistant summarized: "Without P2P, every NCCL all-reduce has to bounce through host RAM, adding latency and halving effective bandwidth." This was the moment when the investigation's frame shifted entirely. The question was no longer "which MoE backend is fastest?" or "how do we tune the server parameters?" but rather "how do we fix the virtualization bottleneck?" The entire tuning effort — CUDA graphs, MoE runner backends, memory fraction adjustments — had been optimizing within a model that assumed the bottleneck was in the GPU software stack. The actual bottleneck was in the virtualization layer, invisible to standard GPU profiling tools.

Part VI: The MoE Kernel Tuning Investigation

Parallel to the virtualization investigation, the assistant explored whether MoE kernel tuning could improve performance. The user's intervention — redirecting the session toward prior research artifacts in a local repository — proved highly productive, even though the configs themselves were not directly applicable.

The repository contained tuning configs generated for a Kimi K2 deployment on the same hardware. The assistant spawned a subagent to analyze these files, discovering that flashinfer's autotuner maintained per-device tuning configuration files — but no config existed for the RTX PRO 6000 Blackwell. The datacenter Blackwell GPUs (B200, GB200) had dedicated tuning files, but the workstation SM120 variant was missing.

The assistant dove deep into flashinfer's autotuner infrastructure, discovering the AutoTuner class, the autotune context manager, and the config loading mechanism. When the autotuner tried to load a config for the RTX PRO 6000, the file path resolved to a non-existent file, causing a fallback to default config 0 with tile_config -1 — meaning no hardware-specific tuning was applied.

However, a critical discovery emerged: the flashinfer_cutlass MoE path — which was the working backend — does not use the flashinfer autotuner at all. The autotuner only covers the flashinfer_trtllm path, which was SM100-only and had already crashed on SM120. The CUTLASS path uses fixed kernels with tune_max_num_tokens as its only configurable parameter.

The assistant tested all viable MoE runner backends — flashinfer_cutlass, flashinfer_cutedsl — and found them all within the same performance band of 195–225 tok/s output throughput. This confirmed that the bottleneck was not in the MoE kernel implementation but in the overall system architecture: the attention mechanism, the all-reduce communication, and the fundamental small-batch decode geometry.

This investigation yielded a valuable compatibility matrix for SM120 GPUs. The flashinfer_trtllm MoE backend is SM100-only and unavailable. The flashmla_kv and flashmla_sparse NSA backends produce NaN on decode. The trtllm NSA backends work correctly. Both flashinfer_cutlass and flashinfer_cutedsl MoE runners work and perform similarly. This matrix is a concrete artifact that will save future engineers significant debugging time when deploying on SM120 hardware.## Part VII: Synthesis and Lessons Learned

The deployment of GLM-5-NVFP4 on eight RTX PRO 6000 Blackwell GPUs traversed a landscape of challenges — from catastrophic NaN crashes to virtualization-induced latency bottlenecks. Each obstacle was met with a combination of systematic experimentation, quantitative analysis, and adaptive tooling. The final throughput of approximately 225 output tokens per second under concurrent load, while modest compared to datacenter configurations, represents a significant achievement given the hardware constraints: PCIe-only interconnect, virtualized environment, and a GPU architecture (SM120) that was not the primary target for the software stack.

More importantly, the session demonstrates a methodology for approaching complex ML deployments. Several enduring lessons emerge from this work.

First, systematic debugging pays off. The NaN crash that consumed the first portion of the session was resolved not through a lucky guess but through methodical elimination of variables. The configuration table — tracking multiple attempts with different combinations of NSA backends, FP8 GEMM backends, KV cache dtypes, and CUDA graph settings — is a model for how to approach complex failure modes where multiple components could be at fault.

Second, measurement infrastructure is not optional. The transition from "the model works" to "the model works well" required overcoming multiple tooling failures — tokenizer resolution issues, reasoning model format incompatibilities, broken per-token metrics. Each failure revealed an assumption baked into the tooling that did not hold for this novel model-hardware combination. The assistant's ability to adapt its measurement strategy rather than getting stuck on perfect instrumentation was essential.

Third, intuition must be validated by arithmetic. The expert parallelism investigation is a case study in the gap between intuitive reasoning and quantitative analysis. The intuition that EP should be faster on PCIe is sound for many models, but the specific parameters of GLM-5-NVFP4 (small hidden size of 6144, aggressive NVFP4 quantization, high expert count of 256) inverted the expected relationship. Only by building a detailed computational model — specifying FLOP counts, communication patterns, bandwidth constraints, and batch effects — could the team discover this counterintuitive result.

Fourth, infrastructure topology is destiny. The ultimate bottleneck was not in the software configuration or kernel implementation but in the hardware topology: a virtualized environment without direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory. This finding emerged only after the assistant had exhausted all software-level tuning options and began questioning the fundamental assumptions about the hardware. The lesson is clear: before optimizing within your current model of the bottleneck, validate that your model is correct.

Fifth, the value of prior research artifacts. The user's intervention to redirect the investigation toward existing tuning configs in a local repository proved highly productive, even though the configs themselves were not directly applicable. The discovery of the tuning methodology, the autotuner infrastructure, and the missing config file for the RTX PRO 6000 all emerged from this redirection. In complex debugging sessions, the most valuable resource is often not new investigation but existing knowledge that has not yet been connected to the current problem.

Conclusion

The complete arc of this segment — from NaN crash to virtualization diagnosis — follows a pattern familiar to any engineer who has debugged complex systems: start with the obvious (the crash), establish a baseline (benchmarks), explore optimizations (tuning, backends), exhaust hypotheses (MoE runners, CUDA graphs, EP), and finally discover the root cause in an unexpected layer (virtualization). The user's question about Proxmox was the critical insight that connected the dots between the puzzling throughput ceiling and the missing P2P support.

For practitioners deploying large models in virtualized environments, the lesson is clear: the virtualization layer is not transparent. Before tuning MoE backends or CUDA graphs, check nvidia-smi topo -p2p. Before concluding the system is compute-bound, measure cross-GPU latency at realistic message sizes. The bottleneck may not be where you think it is — and finding it requires both systematic methodology and the domain-specific intuition to ask the right questions.

The validated configuration that emerged from this work — --nsa-decode-backend trtllm, --nsa-prefill-backend trtllm, --moe-runner-backend flashinfer_cutlass, --attention-backend flashinfer, --fp8-gemm-backend cutlass, --mem-fraction-static 0.92, and --disable-custom-all-reduce — provides a stable foundation for future deployments on SM120 Blackwell GPUs. The baseline performance metrics — ~11 tok/s single-stream, ~210–247 tok/s saturated at 64 concurrent requests — establish a reference point for evaluating future improvements. And the virtualization bottleneck diagnosis opens a clear path forward: either configure direct GPU peer-to-peer access in the Proxmox environment, or accept the latency penalty and design the deployment strategy around it.

As AI models continue to grow in size and complexity, and as the hardware landscape diversifies beyond datacenter-grade infrastructure, the methodology demonstrated in this session — systematic debugging, adaptive measurement, quantitative validation of intuitions, and deep infrastructure analysis — will become increasingly essential. The GLM-5-NVFP4 deployment on 8× RTX PRO 6000 Blackwell GPUs is not just a story about one model on one hardware configuration; it is a case study in how to approach the frontier of ML infrastructure engineering.

References

[1] "Deploying GLM-5-NVFP4 on Blackwell GPUs: From NaN Crashes to Virtualization Bottlenecks" — Chunk article covering the full arc of Segment 2, including NaN crash resolution, baseline benchmarking, server tuning, expert parallelism analysis, MoE kernel investigation, and virtualization diagnosis.

[2] "From NaN to Virtualization: The Complete Arc of Deploying GLM-5-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs" — Chunk article providing a complementary narrative arc through the five phases of the deployment journey.

[3] Analyzer summary for Segment 2 — Provides the thematic framework and high-level narrative of the segment's work.

[4] Subagent session summaries — Detailed analyses of specific sub-investigations including the expert parallelism memory footprint calculation, the flashinfer autotuner infrastructure analysis, and the PCIe bandwidth measurement scripts.