Deploying GLM-5-NVFP4 on Blackwell: From NaN Crashes to Virtualization Bottlenecks

Introduction

Deploying a large Mixture-of-Experts (MoE) model like GLM-5-NVFP4 on cutting-edge Blackwell GPUs is a journey through multiple layers of complexity: hardware compatibility, kernel selection, quantization format support, communication optimization, and infrastructure diagnosis. This article chronicles a concentrated opencode coding session that tackled exactly this challenge — deploying the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs running inside a Proxmox virtualized environment. The session spanned two interconnected threads of work: first, a deep forensic investigation into SGLang's FP4 MoE code paths to understand how the inference engine handles 4-bit quantized MoE computation; and second, a systematic effort to resolve a critical NaN crash during decode, establish baseline throughput, tune server parameters, and diagnose why performance was falling short of expectations.

The story that emerges is one of methodical problem-solving: starting from a crashing server, the team worked through backend selection, benchmarking, parameter tuning, expert parallelism analysis, and ultimately identified virtualization-induced PCIe peer-to-peer latency as the root cause of the throughput bottleneck. Along the way, the investigation produced a comprehensive map of SGLang's FP4 MoE architecture — a valuable artifact for anyone deploying quantized MoE models on this framework.

The FP4 MoE Code Path Investigation

The session began with a subagent tasked with a focused mission: understand the MoE runner code paths in SGLang for NVFP4 (modelopt_fp4) quantization. The server was running with moe_runner_backend='flashinfer_cutlass' and fp4_gemm_runner_backend='flashinfer_cutlass', but the team needed to understand what other backends existed, what the disable_flashinfer_cutlass_moe_fp4_allgather flag controlled, and whether tuning infrastructure was available.

The Six-Pronged Investigation

The user's request was a masterclass in targeted code archaeology [5]. It laid out six concrete lines of inquiry, each designed to peel back a layer of the MoE implementation:

  1. Find the FP4 MoE runner implementation — locate the code that actually executes FP4-quantized MoE layers
  2. Understand the allgather optimization flag — what does disable_flashinfer_cutlass_moe_fp4_allgather control?
  3. Catalog available FP4 backends — what runner backends support FP4 quantization?
  4. Search for tuning scripts — is there auto-tuning infrastructure for the flashinfer_cutlass backend?
  5. Examine the GLM-5 model file — how does the model declare its MoE layers?
  6. Investigate the flashinfer_mxfp4 alternative — could a different backend offer better performance? The assistant executed these probes in parallel, dispatching six SSH commands to the remote machine in a single round [1]. This parallel approach was efficient but also reflected a key constraint of the opencode session protocol: all tool calls in a message are dispatched simultaneously, and the assistant must wait for all results before proceeding.

The Five FP4 MoE Backends

The investigation revealed that SGLang supports five distinct FP4 MoE execution paths, defined in the MoeRunnerBackend enum at utils.py [10]:

| Backend | FP4 Support | Description | |---------|-------------|-------------| | flashinfer_cutlass | Yes (default) | FlashInfer's CUTLASS-based fused MoE kernel | | flashinfer_trtllm | Yes | TensorRT-LLM kernel (SM100-only, requires weight shuffling) | | flashinfer_cutedsl | Yes | CuteDSL-based masked grouped GEMM from FlashInfer | | cutlass | Yes | Standalone sgl_kernel CUTLASS grouped GEMM | | flashinfer_mxfp4 | No (MXFP4 only) | For micro-scaling FP4 quantization, not modelopt_fp4 |

The flashinfer_cutlass backend was the default and the one currently configured. The flashinfer_trtllm path was particularly interesting — it required weight shuffling at load time via align_fp4_moe_weights_for_flashinfer_trtllm(), which reorders rows and shuffles scale matrices to match TensorRT-LLM's expected layout [3]. Critically, this backend was found to be SM100-only, meaning it would not work on the SM120 Blackwell GPUs in the deployment.

The Allgather Optimization

A key finding was the should_use_flashinfer_cutlass_moe_fp4_allgather() function, which controls a communication optimization for DP attention + EP MoE configurations [2]. When enabled (the default), hidden states are quantized to FP4 before the all-gather communication, reducing inter-GPU data transfer by approximately 4x (from 2 bytes per element in bf16 to 0.5 bytes per element in FP4, plus small scale overhead). The flashinfer_cutlass_fused_moe kernel then accepts the pre-quantized FP4 input directly.

When disabled, hidden states are communicated in bf16, and FP4 quantization happens inside the MoE kernel. This flag had implications throughout the GLM-4 MoE model's forward pass — controlling whether shared experts ran with specific TP settings and whether tensor_model_parallel_all_reduce was skipped in favor of reduce-scatter [9].

The Runner Factory and Weight Loading

The investigation traced the runner dispatch logic to the ModelOptNvFp4FusedMoEMethod class in modelopt_quant.py [11]. The create_moe_runner method is the factory that instantiates the appropriate MoE runner based on backend selection. A critical detail emerged in the load_up_proj_weight_first property:

@property
def load_up_proj_weight_first(self) -> bool:
    # FlashInfer CUTLASS kernel assumes [Up, Gate] Proj as W13
    return self.enable_flashinfer_cutlass_moe and self.moe_runner_config.is_gated

This property encodes a subtle but correctness-critical constraint: the FlashInfer CUTLASS kernel expects the fused W13 weight tensor to have the Up projection weights before the Gate projection weights. If weights are loaded in the default [Gate, Up] order, the kernel will silently produce incorrect results. This coupling between backend selection and weight loading order means that changing the backend can silently change how weights must be loaded — a potential source of regressions.

The Tuning Gap

A significant finding was the absence of tuning infrastructure for the flashinfer_cutlass MoE backend [1]. The benchmark directory contained tuning scripts only for the Triton-based fused MoE runner (fused_moe_triton/tuning_fused_moe_triton.py, etc.). The flashinfer_cutlass_fused_moe kernel accepts a tune_max_num_tokens parameter, suggesting FlashInfer handles autotuning internally, but there was no standalone tuning script or benchmark harness for this backend. This meant that if the flashinfer_cutlass backend was underperforming, there was no automated tuning path to improve it.

From Code Investigation to Deployment

The FP4 MoE code path investigation was not an academic exercise — it was driven by the practical need to get GLM-5-NVFP4 running efficiently on the target hardware. The root session had already accomplished the initial deployment, but a critical problem remained: the server was crashing with NaN errors during decode.

Resolving the NaN Crash

The NaN crash turned out to be a backend compatibility issue specific to the SM120 Blackwell GPUs. The solution was to select specific NSA (non-scalar allreduce) backends:

--nsa-decode-backend trtllm --nsa-prefill-backend trtllm

With these flags, the server produced coherent output on the SM120 GPUs. This finding was significant because it demonstrated that not all kernel backends are compatible with all GPU architectures — the flashinfer_trtllm MoE path was SM100-only, and the NSA backends required explicit selection for SM120 compatibility.

Baseline Benchmarking

With the server running, the team established baseline throughput metrics using 64 concurrent requests:

Server Parameter Tuning

The team then experimented with server parameters to improve throughput:

Expert Parallelism Analysis

The user raised the possibility of using expert parallelism (EP) to improve PCIe-bound performance. The assistant analyzed the memory constraints:

Diagnosing the Virtualization Bottleneck

The persistent throughput ceiling led to a critical question: was the Proxmox virtualized environment introducing cross-GPU latency that was limiting performance? The investigation confirmed that the system was a KVM/QEMU VM, and crucially, that there was no direct GPU peer-to-peer support (NS status). This meant all cross-GPU transfers had to pass through host memory via the PCIe bus, rather than using NVIDIA's fast NVLink or P2P direct transfers.

A bandwidth test quantified the impact:

Synthesis: The Full Picture

The work in this session tells a coherent story about the challenges of deploying cutting-edge ML models on virtualized infrastructure. The FP4 MoE code path investigation provided the foundational understanding of how SGLang handles 4-bit quantized MoE computation — the five backends, the allgather optimization, the weight loading constraints, and the tuning gap. This knowledge directly informed the deployment effort: it explained why certain backends worked or didn't work on SM120 GPUs, why the allgather optimization was relevant for the DP attention configuration, and why there was no easy tuning path to better performance.

The deployment effort then built on this understanding, resolving the NaN crash through careful backend selection, establishing baseline metrics, and systematically testing configuration changes. When throughput plateaued, the investigation pivoted to higher-level architectural questions — expert parallelism and virtualization overhead — ultimately identifying the root cause.

The key findings can be summarized as:

  1. FP4 MoE in SGLang is a multi-backend architecture with five distinct execution paths, each with its own weight layout assumptions and hardware compatibility constraints.
  2. The allgather optimization reduces communication volume by 4x by quantizing to FP4 before all-gather, but it's only active under specific conditions (DP attention + EP MoE + flashinfer_cutlass backend).
  3. SM120 GPUs require explicit NSA backend selection (trtllm) to avoid NaN crashes, and the flashinfer_trtllm MoE path is SM100-only.
  4. No tuning infrastructure exists for the flashinfer_cutlass MoE backend, limiting optimization options.
  5. Virtualization-induced P2P latency is the primary performance bottleneck, with small-message all-reduce bandwidth dropping to ~1 GB/s in the Proxmox VM environment.

Implications for Future Work

The diagnosis of virtualization overhead as the primary bottleneck opens several avenues for mitigation:

Conclusion

This session exemplifies the multi-layered nature of ML infrastructure debugging. It began with source code archaeology — tracing through SGLang's MoE runner dispatch, understanding backend selection, and mapping weight loading constraints. It moved into deployment and benchmarking, resolving a NaN crash and establishing baseline metrics. It then ascended to system-level diagnosis, identifying virtualization-induced PCIe latency as the root cause of the throughput bottleneck.

The journey from code to deployment to infrastructure diagnosis required expertise spanning GPU architecture, quantization formats, distributed computing, and virtualization technology. The FP4 MoE code path investigation produced a detailed map of SGLang's inference engine that will serve as a reference for future optimization efforts. And the identification of virtualization overhead as the primary bottleneck provides a clear direction for future work: to unlock the full performance of the Blackwell GPUs, the model must be deployed on infrastructure that supports direct GPU peer-to-peer communication.## References

[1] "Deconstructing the FP4 MoE Code Paths: A Deep Dive into SGLang's NVFP4 Quantization Investigation" — Analysis of the assistant's first response in the FP4 MoE code path investigation, covering the six parallel SSH commands and their results.

[2] "Deep Dive into SGLang's FP4 MoE Runner Investigation: Message 6" — Analysis of the flashinfer_cutlass MoE runner investigation, including the allgather optimization and token dispatcher code.

[3] "Inside SGLang's FP4 MoE Engine: A Deep Dive into the NVFP4 Runner Code Paths" — Detailed examination of the flashinfer_trtllm FP4 runner path, including weight shuffling requirements.

[4] "Completing the Map: Investigating the Third FP4 MoE Backend in SGLang" — Analysis of the flashinfer_cutedsl backend and its masked grouped GEMM approach.

[5] "Navigating the MoE Runner Labyrinth: A Deep Dive into SGLang's FP4 Code Path Investigation" — Analysis of the user's opening message tasking the assistant with the FP4 MoE investigation.

[6] "Deep Dive into SGLang's FP4 MoE Code Paths: Investigating Expert-Parallel MoE and Backend Configuration" — Analysis of the expert-parallel MoE layer and backend configuration.

[7] "Tracing the FP4 MoE Code Path: A Pivot from Layer Code to Quantization Method" — Analysis of the transition from layer code to quantization method investigation.

[8] "Deep Dive into FP4 MoE Code Paths: Systematic Source Code Archaeology in SGLang" — Systematic analysis of the FP4 MoE source code archaeology.

[9] "Tracing the FP4 MoE Configuration: A Deep Dive into SGLang's Server Argument Resolution" — Analysis of how server arguments control FP4 MoE configuration, including the allgather flag.

[10] "Tracing the FP4 MoE Code Paths: A Methodical Deep Dive into SGLang's Runner Dispatch Logic" — Analysis of the runner dispatch logic and backend selection.

[11] "Inside the FP4 MoE Runner Factory: Tracing SGLang's ModelOpt Quantization Dispatch" — Analysis of the ModelOptNvFp4FusedMoEMethod class and the runner factory dispatch.