The Pivot Point: Debugging NaN Crashes in GLM-5-NVFP4 on Blackwell GPUs

Introduction

In the high-stakes world of deploying cutting-edge large language models on novel hardware, few moments are as tense as the one captured in message 209 of this opencode session. The assistant has just launched yet another attempt to serve the GLM-5-NVFP4 model—a quantized Mixture-of-Experts (MoE) model with DeepSeek Sparse Attention (DSA)—across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The server process is alive, the logs show clean startup, and the assistant is waiting, hoping this configuration will finally break the cycle of NaN crashes that have plagued every previous attempt. This message, brief as it appears, represents a critical turning point in an intense debugging session where the assistant synthesizes everything learned from earlier failures into one final, targeted configuration.

The Message in Full

The subject message reads:

[assistant] Now using nsa_prefill_backend='trtllm' and nsa_decode_backend='trtllm'. Still auto-selected fp8_e4m3 KV cache (model-forced). The DeepGemm warning persists but with --fp8-gemm-backend cutlass, it shouldn't use DeepGemm for FP8 ops.

>

Let me wait for loading and test: [bash] sleep 120 && ssh 10.1.230.175 'tail -10 ~/sglang-glm5.log' [2026-02-19 00:21:44 TP7] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [2026-02-19 00:21:46 TP0] max_total_num_tokens=370752, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=64, context_len=202752, available_gpu_mem=17.08 GB [2026-02-19 00:21:46] INFO: Started server process [22711] [2026-02-19 00:21:46] INFO: Waiting for application startup. [2026-02-19 00:21:47] INFO: ...

At first glance, this appears to be a routine status check. But beneath the surface, it encapsulates the entire arc of a debugging journey: the assistant's reasoning about why previous attempts failed, its hypotheses about the root cause, and the specific configuration choices made to address them.

The Context: A History of NaN Crashes

To understand why this message matters, we must trace the debugging trail that led here. The session began with the deployment of GLM-5-NVFP4, a model that uses the novel glm_moe_dsa architecture—a variant of GLM that incorporates DeepSeek's Sparse Attention (DSA). The model is quantized using FP4 (NVFP4 format), meaning weights and activations are stored in 4-bit floating-point format to fit the massive model across the 8 GPUs' combined 768 GB of VRAM.

Every previous attempt to serve this model had ended the same way: a device-side assert triggered error during decode, caused by the probability tensor containing NaN or Inf values. The assistant had systematically explored possible causes:

  1. SM120 shared memory issue: The Blackwell architecture (SM120) introduced new constraints that required a patch in SGLang's attention kernels. The assistant had rebuilt SGLang from the main branch to include PR #14311, which fixed the shared memory problem. Yet the NaN crash persisted.
  2. Attention backend incompatibility: The assistant tried flashinfer, flashmla_sparse, and triton attention backends. All produced the same crash.
  3. DeepGemm scale format mismatch: A persistent warning in the logs stated: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." The assistant initially suspected this was the culprit and tried --fp8-gemm-backend cutlass to bypass DeepGemm entirely.
  4. Transformers 5.2.0 RoPE incompatibility: Another warning flagged that Transformers 5.2.0 might have RoPE parameter incompatibilities with certain models. The assistant briefly downgraded to Transformers 4.57.6 but found that the model's glm_moe_dsa architecture required Transformers 5.2.0+ to be recognized at all.
  5. CUDA graph capture: The assistant tried disabling CUDA graphs, which had helped in previous successful deployments of similar models (documented in the local FINDINGS.md).
  6. KV cache dtype: The model forced fp8_e4m3 KV cache, which in turn forced the NSA attention backends to use flashmla_kv for decode. The assistant tried removing the explicit --kv-cache-dtype flag, only to discover the model auto-selected it anyway.

The Reasoning Behind This Configuration

Message 209 is the culmination of this iterative debugging. The assistant makes three key observations that reveal its refined mental model of the problem:

First observation: "Now using nsa_prefill_backend='trtllm' and nsa_decode_backend='trtllm'." The assistant has explicitly overridden the NSA (Neural Sparse Attention) backends to use trtllm (TensorRT-LLM) instead of the auto-selected flashmla_kv. This is significant because the DSA model's attention path had been identified as a potential source of numerical instability. By switching to trtllm, the assistant is testing whether the NaN originates in the attention computation itself rather than the MoE or GEMM layers.

Second observation: "Still auto-selected fp8_e4m3 KV cache (model-forced)." The assistant has discovered something important: the model configuration itself mandates FP8 KV cache, regardless of what the user passes via --kv-cache-dtype. This means the previous attempt to avoid FP8 KV cache by omitting the flag was futile—the model's architecture definition in Transformers 5.2.0 forces this setting. This is a critical piece of knowledge: the KV cache dtype is not a tunable parameter for this model; it is baked into the architecture.

Third observation: "The DeepGemm warning persists but with --fp8-gemm-backend cutlass, it shouldn't use DeepGemm for FP8 ops." The assistant is reasoning that the DeepGemm warning is a red herring—or at least, that the cutlass backend override should bypass it for the FP8 operations. However, the warning's persistence suggests that DeepGemm might still be engaged for other operations, or that the warning is emitted unconditionally based on model configuration rather than actual backend selection.

Assumptions and Knowledge Boundaries

This message reveals several assumptions the assistant is operating under:

  1. The NaN originates in a specific subsystem: The assistant assumes that by isolating variables—attention backend, GEMM backend, KV cache dtype, CUDA graphs—it can identify the exact source of the numerical instability. This is a reasonable debugging strategy, but it assumes the bug is localized rather than systemic.
  2. The trtllm NSA backend is a viable alternative: The assistant assumes that TensorRT-LLM's attention implementation will handle the DSA model's FP4 quantized attention correctly. This is an untested hypothesis—the previous successful deployment of Kimi K2-Thinking NVFP4 (documented in FINDINGS.md) did not use DSA attention, so there is no precedent for this specific combination.
  3. The model-forced FP8 KV cache is not itself the problem: The assistant accepts the forced fp8_e4m3 KV cache as immutable and works around it rather than trying to patch the model configuration. This is a pragmatic choice, but it leaves open the possibility that the FP8 KV cache quantization itself introduces the NaN.
  4. The DeepGemm warning is a non-issue with cutlass backend: The assistant assumes that setting --fp8-gemm-backend cutlass fully disables DeepGemm. The persistence of the warning suggests this assumption may be incorrect—DeepGemm might be used for non-FP8 operations (e.g., the FP4 matrix multiplications) where the cutlass backend doesn't apply.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. The model forces FP8 KV cache: The assistant discovers and documents that --kv-cache-dtype cannot override the model's internal configuration. This is a significant finding for anyone deploying GLM-5-NVFP4—the KV cache dtype is not a tunable parameter.
  2. NSA backend override is possible: The assistant demonstrates that --nsa-decode-backend and --nsa-prefill-backend can be explicitly set, overriding the auto-selection that would otherwise choose flashmla_kv for FP8 KV cache.
  3. The DeepGemm warning is persistent: Even with --fp8-gemm-backend cutlass, the DeepGemm scale format warning appears. This suggests the warning is emitted based on model configuration analysis rather than actual runtime backend selection, or that DeepGemm is still engaged for some operations.
  4. Server startup is clean: The log output shows that the server initializes successfully, with reasonable memory allocation (17.08 GB available per GPU after loading). The model loads, CUDA graphs are captured, and the server is ready to accept requests. The crash, if it occurs, will happen during decode—not during initialization.

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message is a masterclass in systematic debugging under uncertainty. The thought process proceeds through several layers:

Layer 1 — Configuration audit: The assistant begins by enumerating the current configuration state, noting what is explicitly set and what is auto-selected. This metacognitive step—checking what you know about the system state—is crucial when debugging complex distributed systems where configuration flags interact in unexpected ways.

Layer 2 — Hypothesis refinement: The assistant acknowledges that the DeepGemm warning persists but argues it should be harmless given the cutlass backend. This reveals a refined hypothesis: the NaN is not caused by DeepGemm's scale format mismatch but by something in the attention path. The explicit override of NSA backends to trtllm is the test of this hypothesis.

Layer 3 — Acceptance of constraints: The assistant accepts the model-forced FP8 KV cache rather than fighting it. This is a pragmatic decision—rather than trying to patch the model configuration or downgrade Transformers (which would break model loading), the assistant works within the constraints and adjusts other parameters.

Layer 4 — The wait-and-see posture: The assistant issues a 120-second sleep before checking the log, acknowledging that model loading takes time and that the real test—a decode request—hasn't happened yet. The log output confirms the server started but doesn't show any decode attempts. The assistant is waiting for the moment of truth.

The Broader Significance

Message 209 sits at a pivotal moment in the debugging arc. The assistant has exhausted many obvious avenues: attention backends, GEMM backends, CUDA graphs, KV cache dtype. Each attempt has narrowed the search space. The NaN crash during decode—specifically in the probability tensor—suggests a numerical precision issue in the final softmax or sampling step. By this point, the assistant has learned that:

Conclusion

Message 209 captures a moment of focused debugging where the assistant synthesizes everything learned from a long series of failures into one targeted configuration. The message is deceptively simple—a status check and a wait—but it represents the culmination of an iterative process of hypothesis formation, testing, and refinement. The assistant's reasoning reveals a systematic approach to debugging complex ML infrastructure: isolate variables, accept constraints, test targeted hypotheses, and document findings. Whether this particular configuration succeeds or fails, the knowledge generated in this message—about model-forced KV cache, NSA backend overrides, and the persistence of the DeepGemm warning—advances the collective understanding of deploying FP4-quantized MoE models with sparse attention on Blackwell GPUs. In the world of cutting-edge ML infrastructure, this is how progress is made: one carefully reasoned configuration change at a time.