Deploying GLM-5-NVFP4 on Blackwell GPUs: From NaN Crashes to Virtualization Bottlenecks

Introduction

Deploying a cutting-edge Mixture-of-Experts (MoE) model like GLM-5-NVFP4 on modern hardware is rarely a straightforward affair. When that hardware consists of eight NVIDIA RTX PRO 6000 Blackwell GPUs running inside a Proxmox virtual machine, the challenge multiplies. This article chronicles a critical segment of an opencode coding session where the team moved from a broken deployment (crashing with NaN values during decode) to a working, benchmarked system, and then confronted the deeper performance constraints imposed by their virtualized infrastructure. Along the way, they investigated expert parallelism as a potential optimization, mapped SGLang's parallelism configuration surface, and ultimately identified virtualization-induced PCIe peer-to-peer latency as the dominant bottleneck.

The work described here represents the transition from "making it work" to "making it fast" — a phase where correctness issues are resolved and the focus shifts to understanding and optimizing throughput in a complex, multi-GPU environment.

Resolving the NaN Crash: NSA Backend Selection

The first major hurdle in this segment was a critical correctness issue: the GLM-5-NVFP4 model was producing NaN (Not a Number) values during the decode phase, rendering the deployment unusable. This type of crash is particularly insidious because it doesn't produce an obvious error message — the model simply outputs garbage.

The root cause was traced to the Neural State Attention (NSA) backend selection. SGLang supports multiple NSA backends for different GPU architectures, and the SM120 Blackwell GPUs required specific backends that were not being selected by default. The fix involved explicitly setting two server flags:

• --nsa-decode-backend trtllm
• --nsa-prefill-backend trtllm By selecting the TensorRT-LLM backend for both prefill and decode phases, the NaN crash was resolved and the model began producing coherent output. This was a critical milestone — without it, none of the subsequent performance work would have been possible. This experience underscores a recurring theme in ML infrastructure: GPU architecture-specific code paths matter enormously. The Blackwell SM120 architecture, being relatively new, had not yet been fully tested across all backend combinations, and the default selections were incompatible. The fix required both knowledge of the available backends and an understanding of which ones were appropriate for the target hardware.

Baseline Benchmarking: Measuring Throughput

With the model producing correct output, the next step was to establish baseline performance metrics. The team ran benchmarks with 64 concurrent requests, measuring both output token throughput and total token throughput (including prefill). The results provided a clear starting point:

• Output throughput: ~225 tokens per second
• Total throughput: ~516 tokens per second These numbers served as the baseline against which all subsequent optimizations would be measured. They also provided an early hint that something was amiss — for eight RTX PRO 6000 Blackwell GPUs, these throughput figures were lower than expected, suggesting that a bottleneck existed somewhere in the system.

Server Tuning: Chasing Throughput Gains

With baseline metrics established, the team began a systematic tuning effort. The first parameter adjusted was --mem-fraction-static, which controls how much GPU memory is reserved for the model's static allocations (weights, KV cache, etc.) versus dynamic allocations. Increasing this value from its default to 0.92 allowed the server to pre-allocate more memory, potentially reducing allocation overhead during inference.

The team also enabled CUDA graph capture, a technique that records and replays GPU kernel launches to reduce launch overhead. CUDA graphs are particularly beneficial for workloads with repetitive, predictable computation patterns — which describes LLM inference well. The graph capture succeeded without running out of memory (OOM), which was a positive sign.

However, the throughput improvements were modest. After tuning, the output throughput ranged from approximately 210 to 247 tokens per second — essentially within the noise of the baseline measurement. This was a crucial finding: the bottleneck was not in kernel launch overhead or memory allocation. Something more fundamental was constraining performance.

The team also experimented with different MoE runner backends, testing flashinfer_cutlass and flashinfer_cutedsl variants. These backends implement the MoE computation (gating, expert dispatch, expert computation, combine) using different CUDA kernel strategies. The results were comparable across backends, with no single backend providing a decisive advantage. The flashinfer_trtllm path was found to be SM100-only, ruling it out for the Blackwell GPUs.

Investigating Expert Parallelism: A Code Search Odyssey

At this point, the user raised a strategic question: could expert parallelism (EP) help mitigate the apparent communication bottleneck? In MoE models, expert parallelism distributes different experts across different GPUs, so that each GPU only holds a subset of the total experts. This reduces the memory required per GPU and, in theory, can reduce communication volume because only the activated experts need to communicate during inference — not every layer's activations.

To answer this question, the team launched a subagent investigation into SGLang's source code, searching for expert parallelism support. This investigation, documented across multiple articles [1][2][3][4], was a masterclass in targeted code analysis.

The investigation began with broad grep searches across the SGLang codebase for terms like ep_size, expert_parallel, and ExpertParallel [1]. These searches revealed that EP-related code existed in server_args.py, glm4_moe.py, and several model configuration files. The assistant then drilled deeper, using sed to extract specific argument parser definitions from server_args.py, revealing the exact CLI flag names [2]:

• --expert-parallel-size (aliases: --ep-size, --ep)
• --data-parallel-size (aliases: --dp-size)
• --moe-data-parallel-size (aliases: --moe-dp-size) The investigation also confirmed that the GLM-4 MoE model implementation (glm4_moe.py) fully supports expert parallelism, importing EP infrastructure like get_moe_expert_parallel_world_size and using EP dispatch/combine guards throughout the forward pass [3]. The synthesized findings [4] provided a complete map of SGLang's parallelism landscape, including critical constraints:
• With the flashinfer_cutlass runner backend (required for FP4 quantization), ep_size must be either 1 or equal to tp_size
• When using an A2A backend, ep_size is automatically set to tp_size
• EP can be combined with MoE-specific DP under the constraint ep_size * moe_dp_size == tp_size

The EP Feasibility Analysis: Memory Constraints

Armed with this knowledge, the team analyzed whether expert parallelism would actually help their specific deployment. The analysis revealed a stark memory constraint: the GLM-5-NVFP4 model has approximately 453GB of MoE expert parameters. Each RTX PRO 6000 GPU has 96GB of memory. Full replication of all experts across all GPUs was therefore impossible — the experts alone would require 453GB, leaving no room for KV cache, activations, or other overhead.

Standard EP8 (distributing experts across all 8 GPUs) would place roughly 57GB of experts on each GPU, which is feasible. However, the analysis concluded that EP8 offered no significant advantage over the current TP8 configuration. The reason lies in the model's hidden size: with a small hidden dimension, the all-to-all communication required for expert dispatch and combine has similar volume to the all-reduce communication required for tensor parallelism. EP would not reduce communication enough to make a meaningful difference.

This was a disappointing but important finding. Expert parallelism, while fully supported by both SGLang and the GLM-5 model, was not the solution to the throughput problem.

Diagnosing the Real Bottleneck: Virtualization Overhead

With EP ruled out, the user turned to a more fundamental question: was the virtualized environment itself causing the bottleneck? The system was running inside a Proxmox VM, and cross-GPU communication — essential for both TP and EP — depends on PCIe peer-to-peer (P2P) transfers. In a virtualized environment, these transfers may not be direct.

Investigation confirmed that the system was a KVM/QEMU virtual machine with no direct GPU peer-to-peer support. The NVIDIA Status (NS) output showed that P2P access between GPUs was not supported, forcing all cross-GPU transfers to route through host memory. This adds significant latency compared to direct GPU-to-GPU transfers over NVLink or even native PCIe.

To quantify the impact, the team ran bandwidth tests. The results were revealing:

• Large transfers: ~32 GB/s — reasonable for PCIe Gen4/Gen5
• Small messages (typical of all-reduce): ~1 GB/s — severely degraded This disparity is the smoking gun. LLM inference relies heavily on small all-reduce operations for tensor parallelism — each layer's activations must be synchronized across GPUs. When these small messages are forced through host memory at 1 GB/s instead of direct GPU-to-GPU at 32+ GB/s, the latency penalty is enormous. The throughput ceiling observed in benchmarking (225 tok/s) is a direct consequence of this virtualization-induced communication bottleneck.

Implications and Next Steps

The identification of virtualization overhead as the primary bottleneck reframes the optimization problem. No amount of server tuning (CUDA graphs, memory fractions, MoE backends) can overcome a fundamental infrastructure limitation. The path forward requires either:

1. Moving to bare-metal or passthrough GPU configuration: If the Proxmox host can be configured to pass through the GPUs with native P2P support, the communication bottleneck would be eliminated.
2. Reducing cross-GPU communication: If P2P cannot be enabled, the deployment strategy must minimize communication. This could mean using fewer GPUs per inference instance (e.g., TP=2 or TP=4) and relying on data parallelism to scale throughput across multiple instances.
3. Exploring alternative parallelism strategies: While standard EP offered no advantage, the combination of EP with MoE-specific DP (e.g., TP4 + EP4 + MoE-DP2) might still provide benefits by reducing the all-reduce group size. The comprehensive mapping of SGLang's parallelism options [4] provides the vocabulary for these experiments. The team now knows exactly which flags to set, which constraints apply, and which configurations are valid for their 8-GPU setup.

Conclusion

This segment of the opencode session exemplifies the iterative, diagnostic nature of ML infrastructure engineering. The team progressed through four distinct phases: resolving a correctness crash, establishing baseline performance, attempting systematic tuning, and ultimately identifying the root cause through targeted investigation. Each phase built on the previous one, and the knowledge accumulated along the way — about NSA backends, MoE runner variants, EP constraints, and virtualization behavior — represents a significant investment in understanding the deployment's behavior.

The most valuable outcome is not a faster deployment (yet), but a clear diagnosis of the bottleneck. The virtualization-induced P2P latency explains the throughput ceiling and points toward specific solutions. The EP investigation, while not yielding a silver bullet, produced a detailed map of SGLang's parallelism capabilities that will inform future configuration decisions. And the benchmarking data provides a baseline against which any infrastructure changes can be measured.

In the broader narrative of deploying GLM-5-NVFP4 on Blackwell GPUs, this segment marks the transition from infrastructure setup to performance optimization — and the recognition that sometimes the most important optimization is not in the software stack, but in the hardware configuration underneath it.## References

[1] "Probing SGLang for Expert Parallelism: A Targeted Code Search in an ML Deployment Pipeline" — Analysis of the initial EP investigation request (message 0), examining its motivation, assumptions, and reasoning structure.

[2] "Drilling Down: How a Subagent Mapped SGLang's Expert Parallelism CLI Flags" — Deep dive into the assistant's targeted code search (message 2) that extracted exact CLI flag definitions and constraints from server_args.py.

[3] "Probing SGLang's Expert Parallelism: A Code Search Deep Dive" — Examination of the assistant's first response (message 1), a coordinated set of SSH-based grep searches across the SGLang codebase.

[4] "Mapping the Parallelism Landscape: How SGLang's Expert and Data Parallelism Support Was Discovered for GLM-5 Deployment" — Analysis of the synthesized findings (message 3) that produced the complete flag table, constraint map, and practical configuration recipes.