Deploying GLM-5-NVFP4 on Blackwell GPUs: From NaN Crashes to Virtualization Bottlenecks

Introduction

The deployment of large language models on non-datacenter hardware has become one of the most challenging frontiers in modern ML infrastructure. When a team set out to deploy GLM-5-NVFP4 — a 744-billion-parameter Mixture-of-Experts (MoE) model with DeepSeek Sparse Attention (DSA) — across eight NVIDIA RTX PRO 6000 Blackwell GPUs, they encountered a cascade of obstacles that tested the limits of systematic debugging, performance engineering, and infrastructure analysis. This article synthesizes the work captured in a single chunk of an opencode coding session, tracing the journey from a catastrophic NaN crash during decode, through baseline benchmarking and server tuning, to the ultimate identification of virtualization-induced PCIe latency as the primary performance bottleneck.

The chunk covers approximately 90 messages (indices 211–301) and represents a microcosm of the challenges inherent in deploying cutting-edge AI models on novel hardware configurations. The themes that emerge — systematic hypothesis testing, the gap between intuition and measurement, the importance of understanding hardware topology, and the value of methodical tooling — are broadly applicable to anyone working at the intersection of large models and non-standard GPU infrastructure.

Part I: The NaN Crisis and the Breakthrough with trtllm NSA Backends

The session's central blocker was a catastrophic failure mode: every time the model attempted to generate tokens beyond the initial prefill, the server crashed with a CUDA device-side assert: "probability tensor contains either inf, nan or element < 0." This NaN (Not-a-Number) error during decode was not a random glitch — it was a systematic failure that persisted across six different configuration attempts, each carefully documented in a configuration table [1].

The root cause lay in the interaction between the model's DSA attention mechanism and the SM120 GPU architecture. GLM-5-NVFP4 uses DeepSeek Sparse Attention, which forces SGLang to use specialized NSA (Native Sparse Attention) backends. The available backends — flashmla_kv, flashmla_sparse, and trtllm — implement the same mathematical operation but through different CUDA kernel paths. On the RTX PRO 6000's SM120 architecture (which has only 101KB of shared memory compared to 228KB on datacenter SM100 Blackwell), the flashmla_kv and flashmla_sparse backends silently produced numerical instability [5].

The breakthrough came in message 215, when the assistant launched the server with --nsa-decode-backend trtllm --nsa-prefill-backend trtllm and received the first coherent output. The trtllm backend, backed by NVIDIA's TensorRT-LLM library, handled the SM120 architecture correctly where the flashmla-based implementations failed. This was confirmed through progressively more demanding tests: a simple "What is 2+2?" query, a longer generation with 256 tokens, and a code-generation task requesting a Python is_prime function — all produced correct, well-formatted output [6][7][8].

The debugging methodology that led to this fix is worth examining. The assistant maintained a structured table of seven configuration attempts, each isolating one or two variables. This systematic elimination of hypotheses — ruling out DeepGemm, CUDA graphs, KV cache dtype, and attention backends in turn — is textbook scientific debugging applied to ML infrastructure. The key insight was recognizing that the model card's recommended configuration, designed for datacenter SM100 Blackwell, was not applicable to the SM120 workstation variant [1].

Part II: Establishing Baseline Throughput and Overcoming Tooling Friction

With the NaN crash resolved, the assistant pivoted to performance measurement. This transition from "does it work?" to "how fast does it work?" proved surprisingly challenging, as the standard benchmarking tools were not designed for reasoning models [9][10].

The first benchmark attempt (message 221) crashed with a 401 Unauthorized error because the bench_serving tool conflated the served model name (glm-5) with the HuggingFace repository identifier (lukealonso/GLM-5-NVFP4) [11]. The second attempt (message 222) corrected this with a --tokenizer flag but then crashed during metrics calculation because the reasoning model returns content: null during the thinking phase, and the tokenizer could not handle a null generated_text [12]. The third attempt (message 223) switched to the native SGLang backend, which worked but produced broken per-token metrics because the non-streaming /generate endpoint cannot provide token-level timing [13].

The assistant's response to these failures is instructive. Rather than trying to fix the benchmarking tool (a major engineering detour), it adapted its measurement strategy to work within the tool's limitations. It identified aggregate output token throughput as the one reliable metric and used that as its north star [14]. The first meaningful baseline — 144 output tok/s and 304 total tok/s with 32 concurrent requests — was established in message 225, with the assistant explicitly noting that "All 32 requests succeeded — no crashes under concurrent load" [15].

Subsequent benchmarks with 64 concurrent requests pushed throughput to approximately 225 output tok/s and 516 total tok/s [16][17][18]. This became the reference point against which all tuning efforts would be measured.

Part III: Server Tuning and the PCIe Bottleneck Revelation

With a baseline established, the assistant systematically explored server tuning parameters. Increasing --mem-fraction-static from 0.85 to 0.92 expanded the KV cache from 370K to 498K tokens without OOM [24][25][26]. Enabling CUDA graphs — a GPU optimization that pre-compiles kernel launch sequences — captured successfully for batch sizes up to 64 [27][28].

But the results were disappointing. Despite successful CUDA graph capture, throughput remained stubbornly in the 210–247 tok/s range — a mere 5% improvement over the baseline [30]. This negative result was itself diagnostic: if CUDA graphs (which specifically optimize kernel launch overhead) did not help, then the bottleneck must be in the computation or communication that happens between kernel launches, not in the launches themselves.

The assistant articulated the conclusion clearly in message 240: "The throughput is capped around 200-236 tok/s regardless of CUDA graphs. This suggests the bottleneck is the MoE expert computation + all-reduce over PCIe, not kernel launch overhead" [30]. This marked a critical shift in mental model — from a "kernel launch overhead" frame to a "communication-bound" frame.

The single-stream latency test confirmed the severity of the bottleneck: approximately 11 tokens per second for a single request [31][32]. With 61 MoE layers and 8 GPUs performing all-reduce after each layer, the PCIe interconnect was being hammered by small messages that could not saturate the theoretical bandwidth but were latency-limited by the round-trip time through the host bridge.

Part IV: Expert Parallelism — When Intuition Meets Arithmetic

The user, recognizing the PCIe bottleneck, posed a provocative question in message 243: "Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?" [33] This launched a deep investigation into Expert Parallelism (EP) as an alternative to Tensor Parallelism (TP).

The initial analysis was promising. EP replaces TP's all-reduce (which requires every GPU to synchronize after every layer) with all-to-all communication (which only requires communication for token routing). For MoE models with large hidden dimensions, this can dramatically reduce cross-GPU traffic [34][35][36].

But the user went further, proposing in message 247 an even more elegant idea: replicate all 256 experts on every GPU, eliminating cross-GPU communication for expert computation entirely [37]. Each GPU would independently compute its assigned expert activations using locally stored weights, with only the initial dispatch and final collection requiring communication.

The arithmetic, however, was unforgiving. The assistant wrote a comprehensive Python script (message 250) that computed the exact memory footprint of every model component [40]. The results were stark:

Part V: The Virtualization Revelation

The investigation took another turn when the assistant began measuring actual PCIe throughput. The results were shocking: large transfers achieved approximately 32 GB/s (close to the theoretical PCIe Gen5 x16 limit), but small messages — the kind used in all-reduce operations — achieved only about 1 GB/s [49][50].

This discrepancy pointed to a deeper issue. The assistant checked the GPU topology with nvidia-smi topo -m and discovered that all eight GPUs were connected via PHB (PCIe Host Bridge) links, with no direct GPU peer-to-peer (P2P) support. The P2P Status showed "NS" (Not Supported) for all GPU pairs [48][50].

Further investigation confirmed that the system was a KVM/QEMU virtual machine running under Proxmox. In such an environment, the hypervisor does not expose direct GPU P2P DMA capabilities to the guest VM. All cross-GPU transfers must traverse the host memory — going from GPU → host memory via PCIe, then from host memory → destination GPU via another PCIe hop. This doubles the PCIe traversal for every cross-GPU communication and, more importantly, introduces the latency of two host memory copies [52][53].

The bandwidth test results told the story: ~32 GB/s for large transfers (which can amortize the per-transfer overhead) but only ~1 GB/s for 4KB messages typical of all-reduce (where the per-transfer latency dominates). The virtualization layer was adding significant latency to every small cross-GPU communication, and with 61 MoE layers each requiring an all-reduce, this latency multiplied into a severe throughput bottleneck [54][55].

Part VI: The MoE Kernel Tuning Investigation

Parallel to the virtualization investigation, the assistant explored whether MoE kernel tuning could improve performance. The user's intervention in message 270 — "./ did some weird tuning configs, use an expert to figure out what that was exactly @README.md" — redirected the session toward prior research artifacts in a local repository [60].

The repository contained tuning configs generated for a Kimi K2 deployment on the same hardware. The assistant spawned a subagent to analyze these files, discovering that flashinfer's autotuner maintained per-device tuning configuration files — but no config existed for the RTX PRO 6000 Blackwell [62][63]. The datacenter Blackwell GPUs (B200, GB200) had dedicated tuning files, but the workstation SM120 variant was missing.

The assistant dove deep into flashinfer's autotuner infrastructure [72][73][74][75]. It discovered the AutoTuner class, the autotune context manager, and the config loading mechanism. When the autotuner tried to load a config for the RTX PRO 6000, the file path resolved to a non-existent file, causing a fallback to default config 0 with tile_config -1 — meaning no hardware-specific tuning was applied [76][77].

However, a critical discovery emerged: the flashinfer_cutlass MoE path — which was the working backend — does not use the flashinfer autotuner at all. The autotuner only covers the flashinfer_trtllm path, which was SM100-only and had already crashed on SM120 [86][87]. The CUTLASS path uses fixed kernels with tune_max_num_tokens as its only configurable parameter.

The assistant tested all viable MoE runner backends — flashinfer_cutlass, flashinfer_cutedsl — and found them all within the same performance band of 195–225 tok/s output throughput [88][89][90]. This confirmed that the bottleneck was not in the MoE kernel implementation but in the overall system architecture: the attention mechanism, the all-reduce communication, and the fundamental small-batch decode geometry.

Part VII: Synthesis and Lessons Learned

The chunk of work captured in this session offers several enduring lessons for ML infrastructure engineering.

First, systematic debugging pays off. The NaN crash that consumed the first portion of the session was resolved not through a lucky guess but through methodical elimination of variables. The configuration table — tracking seven attempts with different combinations of NSA backends, FP8 GEMM backends, KV cache dtypes, and CUDA graph settings — is a model for how to approach complex failure modes where multiple components could be at fault.

Second, measurement infrastructure is not optional. The transition from "the model works" to "the model works well" required overcoming multiple tooling failures — tokenizer resolution issues, reasoning model format incompatibilities, broken per-token metrics. Each failure revealed an assumption baked into the tooling that did not hold for this novel model-hardware combination. The assistant's ability to adapt its measurement strategy rather than getting stuck on perfect instrumentation was essential.

Third, intuition must be validated by arithmetic. The expert parallelism investigation is a case study in the gap between intuitive reasoning and quantitative analysis. The intuition that EP should be faster on PCIe is sound for many models, but the specific parameters of GLM-5-NVFP4 (small hidden size of 6144, aggressive NVFP4 quantization, high expert count of 256) inverted the expected relationship. Only by building a detailed computational model — specifying FLOP counts, communication patterns, bandwidth constraints, and batch effects — could the team discover this counterintuitive result.

Fourth, infrastructure topology is destiny. The ultimate bottleneck was not in the software configuration or kernel implementation but in the hardware topology: a virtualized environment without direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory. This finding emerged only after the assistant had exhausted all software-level tuning options and began questioning the fundamental assumptions about the hardware. The lesson is clear: before optimizing within your current model of the bottleneck, validate that your model is correct.

Fifth, the value of prior research artifacts. The user's intervention to redirect the investigation toward existing tuning configs in a local repository proved highly productive, even though the configs themselves were not directly applicable. The discovery of the tuning methodology, the autotuner infrastructure, and the missing config file for the RTX PRO 6000 all emerged from this redirection. In complex debugging sessions, the most valuable resource is often not new investigation but existing knowledge that has not yet been connected to the current problem.

Conclusion

The deployment of GLM-5-NVFP4 on eight RTX PRO 6000 Blackwell GPUs traversed a landscape of challenges — from catastrophic NaN crashes to virtualization-induced latency bottlenecks. Each obstacle was met with a combination of systematic experimentation, quantitative analysis, and adaptive tooling. The final throughput of approximately 225 output tokens per second under concurrent load, while modest compared to datacenter configurations, represents a significant achievement given the hardware constraints: PCIe-only interconnect, virtualized environment, and a GPU architecture (SM120) that was not the primary target for the software stack.

More importantly, the session demonstrates a methodology for approaching complex ML deployments. The assistant's willingness to build detailed computational models, to question its own assumptions, to adapt its measurement strategy when tools fail, and to systematically eliminate hypotheses before pivoting to new ones — these are skills that transcend the specific hardware and model configuration. As AI models continue to grow in size and complexity, and as the hardware landscape diversifies beyond datacenter-grade infrastructure, this methodology will become increasingly essential.## References

[1] "The Moment of Reckoning: A Systematic Debugging Chronicle for GLM-5-NVFP4 on Blackwell SM120 GPUs" — Message 211 article documenting the seven-attempt configuration table and root cause analysis.

[5] "The Breakthrough: How trtllm NSA Backends Rescued GLM-5-NVFP4 on SM120 Blackwell GPUs" — Message 215 article describing the first successful inference test.

[6] "The Breakthrough: How trtllm NSA Backends Rescued GLM-5-NVFP4 on SM120 Blackwell GPUs" — Message 216 article confirming coherent output with longer generation.

[7] "The Breakthrough: GLM-5-NVFP4 Comes Alive on SM120 Blackwell" — Message 217 article validating code generation correctness.

[8] "The Breakthrough: How trtllm NSA Backends Unlocked GLM-5-NVFP4 on Blackwell SM120 GPUs" — Message 218 article summarizing the breakthrough and baseline throughput.

[9] "The Calm Before the Benchmark: A Methodical Transition from Debugging to Performance Tuning" — Message 219 article on the pivot to benchmarking.

[10] "From Crisis to Calibration: The Pivot to Performance Benchmarking in Message 220" — Message 220 article on establishing baseline methodology.

[11] "The First Benchmark That Never Ran: A Case Study in Tool Assumptions" — Message 221 article on the 401 tokenizer error.

[12] "The Tokenizer Trap: Debugging Benchmark Tooling for a Reasoning Model on Blackwell GPUs" — Message 222 article on reasoning model format incompatibility.

[13] "The Benchmark That Broke: Diagnosing Reasoning Model Compatibility in SGLang's bench_serving" — Message 223 article on the native backend workaround.

[14] "Reading the Signal Through the Noise: Benchmarking GLM-5-NVFP4 After the NaN Crisis" — Message 224 article on adapting measurement strategy.

[15] "Establishing Baseline Throughput: The Moment GLM-5-NVFP4 Proved Stable Under Concurrent Load" — Message 225 article on the first meaningful baseline.

[16] "Benchmarking a 744B MoE on Blackwell GPUs: The Methodical Pursuit of Throughput" — Message 226 article.

[17] "Benchmarking GLM-5-NVFP4 on 8 Blackwell GPUs: Finding the Throughput Ceiling" — Message 227 article.

[18] "The Saturation Point: Benchmarking GLM-5-NVFP4 on Eight Blackwell GPUs" — Message 228 article on 64-concurrent-request benchmarks.

[24] "The Turning Point: Enabling CUDA Graphs for GLM-5-NVFP4 on Blackwell GPUs" — Message 234 article on CUDA graph capture.

[25] "The Weight-Loading Check: A Pivotal Moment in Server Tuning" — Message 235 article.

[26] "The Moment of Truth: Waiting for CUDA Graphs on 8 Blackwell GPUs" — Message 236 article.

[27] "The CUDA Graph Milestone: Validating Inference Stability on SM120 with GLM-5-NVFP4" — Message 237 article.

[28] "The Moment of Validation: CUDA Graphs Go Live on GLM-5-NVFP4" — Message 238 article.

[30] "The Plateau: Recognizing the PCIe Bottleneck in Blackwell MoE Inference" — Message 240 article on the CUDA graph plateau and PCIe bottleneck diagnosis.

[31] "The Single-Stream Latency Test: Understanding the Performance Ceiling of GLM-5-NVFP4 on Blackwell GPUs" — Message 241 article.

[32] "The 11 Tok/s Wall: Diagnosing PCIe-Bound MoE Throughput on 8× Blackwell GPUs" — Message 242 article.

[33] "The Pivot Question: When a User Asks 'Would Expert-Parallel Be Faster?'" — Message 243 article on the user's EP question.

[34] "Expert Parallel vs Tensor Parallel: A Critical Decision for PCIe-Bound MoE Inference" — Message 244 article.

[35] "The Pivot Point: Evaluating Expert Parallelism for PCIe-Bound Blackwell Inference" — Message 245 article.

[36] "Investigating Expert Parallelism for Blackwell GPUs on PCIe: A Diagnostic Deep Dive" — Message 246 article.

[37] "When Intuition Meets Memory Physics: A User's Expert Replication Proposal for GLM-5 on 8 Blackwell GPUs" — Message 247 article.

[40] "The Memory Wall: Why You Cannot Replicate 453 GB of MoE Experts Across 8 GPUs" — Message 250 article on the memory footprint analysis.

[41] "The Communication Calculus: Why Expert Parallelism Fails on PCIe-Bound Blackwell GPUs" — Message 251 article.

[42] "The Latency Mirage: When Expert Parallelism Meets PCIe Reality on Blackwell GPUs" — Message 252 article on the computational model.

[48] "The Turning Point: When a Debugging Session Realized PCIe Wasn't the Bottleneck" — Message 257 article.

[49] "Diagnosing the PCIe Bottleneck: A Deep Dive into GPU Topology Verification" — Message 258 article.

[50] "The Pivot Point: How a Single nvidia-smi Command Unraveled the PCIe Bottleneck Myth" — Message 260 article on the topology discovery.

[52] "The Power of Three Words: Deconstructing 'fix script' in an Expert Debugging Session" — Message 262 article.

[53] "The Minimal Fix: A Single Bash Command That Unlocks Performance Debugging" — Message 263 article.

[54] "The Profiling Loop That Wasn't: A Microcosm of Debugging Blackwell Inference Performance" — Message 264 article.

[55] "The Power of 'Nope': A Single-Word Correction That Reshaped a Debugging Session" — Message 265 article.

[60] "The Cryptic Pivot: How a Three-Word Message Redirected a GPU Inference Debugging Session" — Message 270 article on the user's intervention.

[62] "Delegating Analysis: The Strategic Use of Subagents in MoE Configuration Investigation" — Message 272 article.

[63] "The MoE Kernel Tuning Hypothesis: A Pivot in Debugging GLM-5-NVFP4 Throughput" — Message 273 article.

[72] "Probing the Kernel Frontier: Investigating Autotune Mechanisms for Blackwell SM120 MoE Execution" — Message 282 article.

[73] "Discovering SM120-Specific CUTLASS MoE Kernels: A Turning Point in Blackwell GPU Optimization" — Message 283 article.

[74] "Unearthing the Autotuner: A Pivotal Discovery in Kernel Optimization for Blackwell GPUs" — Message 284 article.

[75] "The Missing Config: Discovering FlashInfer's Autotuner Gap for Blackwell RTX PRO 6000" — Message 285 article.

[76] "The Missing Config: How a Single File Explained 45% of GPU Performance" — Message 286 article.

[77] "Unlocking Blackwell Performance: The Hunt for MoE Kernel Tuning Configs in FlashInfer" — Message 287 article.

[86] "The Autotuner That Wasn't: A Pivotal Discovery in GPU Kernel Tuning" — Message 296 article on the discovery that CUTLASS doesn't use the autotuner.

[87] "The Autotuner Mirage: A Diagnostic Pivot in Blackwell MoE Inference" — Message 297 article.

[88] "The Last MoE Backend: Systematic Kernel Exploration on Blackwell GPUs" — Message 298 article.

[89] "Verifying Flashinfer CuteDSL Backend: The Final Backend Test on Blackwell SM120 GPUs" — Message 299 article.

[90] "Testing the CuteDSL MoE Backend: A Moment of Uncertainty in the GLM-5-NVFP4 Deployment" — Message 300 article on the CuteDSL backend test.