The Sixth Attempt: Debugging NaN Crashes in GLM-5-NVFP4 on Blackwell GPUs

Introduction

In the high-stakes world of deploying cutting-edge large language models on new GPU architectures, success rarely comes from a single command. It emerges from a process of iterative hypothesis testing, where each failure narrows the space of possible causes. Message 204 in this opencode session captures a pivotal moment in such a debugging odyssey: the sixth attempt to launch the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang, after five previous attempts all ended in the same silent crash — a device-side assert triggered error caused by NaN (Not-a-Number) values in the probability tensor during decode.

This article examines that single message in depth: the reasoning that motivated it, the decisions embedded in its command-line flags, the assumptions — both correct and incorrect — that shaped its design, and the broader context of deploying quantized Mixture-of-Experts models on the bleeding edge of GPU hardware.

The Context: A Pattern of Failure

Before examining message 204 itself, it is essential to understand what preceded it. The assistant had been locked in a multi-round debugging battle. The GLM-5-NVFP4 model — a 4-bit NVFP4-quantized variant of GLM-5 using the glm_moe_dsa (DeepSeek Sparse Attention) architecture — had been loaded successfully onto eight Blackwell GPUs with tensor parallelism. The model weights loaded, the CUDA graphs were captured, the prefill warmup passed. But every single time a real inference request came in, the server crashed during decode with the same error: an assertion that the probability tensor contained inf, nan, or elements less than zero.

The assistant had tried multiple configurations. In [msg 178], it used --nsa-decode-backend flashmla_sparse. In [msg 187], it switched to --fp8-gemm-backend cutlass to avoid the DeepGemm library, which had been emitting warnings about an incompatible scale format (ue8m0). Both attempts crashed identically. The error was consistent and reproducible — the model could prefill but could not decode a single token.

A critical turning point came when the user instructed the assistant to read the local research repository (FINDINGS.md) in [msg 188]. This repository documented previous successful deployments of the Kimi K2-Thinking NVFP4 model on the exact same hardware. The findings revealed that the earlier deployment had used --disable-cuda-graph and had not needed to override the FP8 GEMM backend. The key difference was architectural: GLM-5 uses DSA (DeepSeek Sparse Attention), which forces the use of NSA (Non-Self-Attention) backends that the Kimi K2 model did not require.

The Message: A New Hypothesis

Message 204 opens with the assistant articulating a refined hypothesis:

"Now let me try a different approach. The NaN during decode could also be from the DSA attention itself. Let me try with --disable-cuda-graph (which the previous K2-Thinking runs used successfully) and also check if we can set --nsa-decode-backend trtllm or avoid fp8 kv cache."

This sentence reveals the assistant's evolving mental model. Earlier attempts had focused on the GEMM (general matrix multiply) path — first trying different attention backends, then forcing cutlass for FP8 GEMM to avoid DeepGemm's scale format issue. But the persistence of the NaN crash even after switching to cutlass suggested that the GEMM path might not be the culprit. The assistant now considers an alternative: the DSA attention mechanism itself might be producing the NaN values.

The reasoning is sound. DSA (DeepSeek Sparse Attention) is a specialized attention mechanism designed for efficient long-context inference. On Blackwell (SM120) GPUs, SGLang's implementation of DSA uses NSA backends — specialized CUDA kernels for sparse attention computation. If these kernels have a numerical bug on SM120, or if they interact badly with the FP4-quantized weights or KV cache, the result would be NaN logits during decode. This is a fundamentally different failure mode from the DeepGemm scale format issue that the assistant had been chasing.

The Command: A Carefully Crafted Configuration

The bash command that follows is not a random collection of flags. It is a carefully curated set of parameters, each representing a decision based on prior failures:

python3 -m sglang.launch_server \
  --model lukealonso/GLM-5-NVFP4 \
  --served-model-name glm-5 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --trust-remote-code \
  --tp 8 \
  --mem-fraction-static 0.88 \
  --max-running-requests 64 \
  --quantization modelopt_fp4 \
  --attention-backend flashinfer \
  --fp8-gemm-backend cutlass \
  --moe-runner-backend flashinfer_cutlass \
  --disable-custom-all-reduce \
  --enable-flashinfer-allreduce-fusion \
  --disable-cuda-graph \
  --host 0.0.0.0 \
  --port 8000

Let us examine each significant flag and the reasoning behind it:

--quantization modelopt_fp4: This flag tells SGLang to use NVIDIA's Model Optimizer FP4 quantization path. This is the correct quantization method for the GLM-5-NVFP4 checkpoint, which was quantized using NVIDIA's Model Optimizer toolkit. The assistant has used this consistently across all attempts.

--attention-backend flashinfer: FlashInfer is a library of high-performance attention kernels. The assistant has stuck with this backend throughout, as it is the recommended backend for Blackwell GPUs and was used successfully in the Kimi K2 deployment.

--fp8-gemm-backend cutlass: This is a carryover from the previous attempt ([msg 187]). The assistant forced the FP8 GEMM backend to cutlass (a CUDA template library for matrix operations) rather than leaving it on auto, which would select DeepGemm on Blackwell. The DeepGemm warning about incompatible scale format (ue8m0) was the assistant's leading hypothesis for the NaN issue at that point.

--moe-runner-backend flashinfer_cutlass: This specifies the MoE (Mixture-of-Experts) runner backend. The assistant has used this consistently since the first attempt after reading the HuggingFace model card's recommendations.

--disable-cuda-graph: This is the key new flag in this attempt. CUDA graphs allow the GPU to capture and replay sequences of kernel launches with minimal CPU overhead, improving performance. However, the previous Kimi K2-Thinking deployment had used --disable-cuda-graph successfully. The assistant's hypothesis is that CUDA graph capture might be introducing numerical instability in the DSA attention path — perhaps by caching incorrect memory addresses or by replaying kernels with stale data.

Notably absent: There is no --kv-cache-dtype fp8_e4m3 flag, which was present in earlier attempts. By omitting it, the KV cache dtype defaults to auto, which may select a different format. There is also no --nsa-decode-backend override, meaning the NSA backend will use its default selection.

The Thinking Process: What the Assistant Got Right

The assistant's reasoning in this message demonstrates several strengths:

  1. Integration of local research: After being directed to the local repository in [msg 188], the assistant read FINDINGS.md and extracted actionable information. The finding that the Kimi K2 deployment used --disable-cuda-graph was directly applied to this attempt.
  2. Hypothesis refinement: The assistant correctly recognized that the DeepGemm scale format hypothesis might be incomplete. If forcing cutlass didn't fix the NaN, the problem might lie elsewhere — specifically in the DSA attention path that distinguishes GLM-5 from the previously successful Kimi K2 deployment.
  3. Conservative modification: Rather than changing many flags at once, the assistant made a single significant change (adding --disable-cuda-graph) while keeping other flags from the previous attempt. This is good debugging practice — isolate one variable at a time.
  4. Environmental awareness: The assistant maintained the correct environment variables (CUDA_HOME, NCCL_* settings, OMP_NUM_THREADS) across all attempts, ensuring that the execution environment remained consistent.

Assumptions and Potential Blind Spots

Despite the sound reasoning, the message rests on several assumptions that may not hold:

Assumption 1: The NaN is caused by the DSA attention path. This is the central hypothesis of this attempt, but it is far from proven. The NaN could still originate from the GEMM path despite the cutlass override — perhaps the FP4 GEMM (as opposed to FP8 GEMM) has its own numerical issues on SM120. The --fp8-gemm-backend flag controls FP8 GEMM, but the FP4 GEMM path (used for the actual quantized weights) might use a different backend.

Assumption 2: Disabling CUDA graphs will fix the DSA attention issue. The Kimi K2 deployment used --disable-cuda-graph, but that model used a different architecture (KimiK25) without DSA. The reason the Kimi K2 deployment needed --disable-cuda-graph might have been unrelated to attention — it could have been a workaround for a different kernel incompatibility. Applying the same fix to GLM-5 assumes the root cause is the same.

Assumption 3: The DeepGemm scale format warning is not the primary cause. By keeping --fp8-gemm-backend cutlass but not addressing the DeepGemm issue more fundamentally, the assistant assumes that the cutlass backend avoids the problem entirely. However, if the checkpoint's scale format is fundamentally incompatible with all FP8 GEMM backends on Blackwell, the NaN might persist regardless of which backend is selected.

Assumption 4: The Transformers 5.2.0 RoPE incompatibility is not the issue. Earlier in the conversation ([msg 184]), the assistant noted a warning about potential RoPE parameter incompatibilities with Transformers >=5.0.0. The assistant had even briefly downgraded to Transformers 4.57.6 ([msg 201]) but found that the model required 5.2.0 for architecture registration. In this attempt, the assistant has not addressed the RoPE warning, implicitly assuming it is not the source of the NaN.

Input Knowledge Required

To fully understand this message, a reader would need knowledge of:

  1. SGLang's architecture and flag system: Understanding what --disable-cuda-graph, --fp8-gemm-backend, --nsa-decode-backend, and --moe-runner-backend control requires familiarity with SGLang's serving stack and its modular backend system.
  2. Blackwell (SM120) GPU architecture: The message is deeply shaped by the specifics of NVIDIA's Blackwell architecture — the DeepGemm library's behavior on SM120, the shared memory constraints that required PR #14311, and the NSA backend compatibility matrix.
  3. NVFP4 quantization: The modelopt_fp4 quantization method and its interaction with different GEMM backends is critical context. The scale format warning (ue8m0) refers to the format used for quantization scales in the checkpoint.
  4. DSA (DeepSeek Sparse Attention): Understanding that GLM-5 uses DSA, which forces NSA backends, and how this differs from standard attention mechanisms, is essential to following the assistant's hypothesis.
  5. The previous Kimi K2 deployment: The FINDINGS.md document that the assistant consulted documented a successful deployment of a different NVFP4 model on the same hardware. The differences between that deployment and the current one are central to the assistant's reasoning.

Output Knowledge Created

This message creates several pieces of knowledge:

  1. A testable configuration: The specific combination of flags in this message represents a hypothesis that can be tested. If the server starts successfully and handles inference requests without NaN crashes, it would confirm that CUDA graph capture was the source of numerical instability. If it crashes again, the hypothesis is falsified.
  2. A documented debugging step: The message adds to the iterative record of what has been tried and what has failed. This is valuable for future debugging — if this attempt also fails, the assistant can rule out both the DeepGemm scale format issue (addressed in attempt 5) and the CUDA graph capture issue (addressed in attempt 6).
  3. A refined problem space: By shifting the hypothesis from "DeepGemm scale format incompatibility" to "DSA attention numerical instability," the message narrows the search space. If this attempt fails, the next logical step would be to investigate the DSA attention kernels more directly — perhaps by trying different NSA backends (trtllm, flashmla_sparse) or by examining the RoPE parameter compatibility issue.

The Broader Significance

Message 204 is a microcosm of the challenges involved in deploying state-of-the-art AI models on new hardware. The GLM-5-NVFP4 model represents the cutting edge of both model architecture (quantized MoE with sparse attention) and hardware (Blackwell GPUs). When these two cutting edges meet, the failure modes are often subtle and difficult to diagnose.

The assistant's approach — iterative hypothesis testing informed by local research, with careful isolation of variables — is a textbook example of systematic debugging. But the message also reveals the limits of this approach. Each attempt takes 3-5 minutes (model loading, warmup, and crash), and with six attempts already made, the debugging process is consuming significant time. The assistant has not yet considered some alternative approaches, such as:

Conclusion

Message 204 captures a moment of hypothesis refinement in a complex debugging session. The assistant, having exhausted the DeepGemm scale format hypothesis, pivots to investigating the DSA attention path as the source of NaN values during decode. The addition of --disable-cuda-graph — inspired by a successful previous deployment of a different model on the same hardware — represents a targeted test of this new hypothesis.

Whether this attempt succeeds or fails, the message demonstrates the disciplined approach required to deploy cutting-edge AI systems: form a hypothesis, design a test, execute it, and let the evidence guide the next step. In the world of Blackwell GPUs and quantized MoE models, there are no silver bullets — only careful, iterative debugging.