The Moment of False Hope: Debugging NaN Crashes on Blackwell

In the middle of a grueling debugging session spanning hours, message [msg 195] arrives as a moment of cautious optimism—one that would be swiftly shattered. The assistant writes:

Server is up and warmup passed. Let me test:

``bash ssh 10.1.230.175 'curl -s --max-time 120 http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d "{\"model\": \"glm-5\", \"messages\": [{\"role\": \"user\", \"content\": \"What is 2+2? Answer briefly.\"}], \"max_tokens\": 50}" 2>&1' ``

This message is deceptively simple. It contains no reasoning block, no analysis, no complex decision-making—just a single curl command to test a freshly launched server. Yet its placement in the conversation makes it a critical inflection point. To understand why this message matters, we must examine the long chain of failures that preceded it and the crushing discovery that followed.

The Context: A Cascade of Crashes

The assistant had been attempting to deploy the GLM-5-NVFP4 model—a massive Mixture-of-Experts (MoE) language model with DeepSeek Sparse Attention (DSA)—on eight NVIDIA RTX PRO 6000 Blackwell GPUs using the SGLang inference framework. This was not a straightforward deployment. The Blackwell architecture (SM120) was still new, and SGLang's support for it was evolving rapidly, requiring a build from the main branch to incorporate a critical shared memory fix (PR #14311).

Every prior attempt had ended in failure. The server would load the model, capture CUDA graphs during warmup, and appear to start successfully. But the moment a real decoding request arrived, it would crash with a device-side assert triggered error. The root cause, as the assistant had painstakingly traced in [msg 186], was a NaN/Inf value in the probability tensor during the decode phase. The error message was explicit: "Assertion probability tensor contains either inf, nan or element < 0 failed."

The assistant had already explored multiple hypotheses. It tried switching attention backends from flashinfer to triton ([msg 170]), only to hit an assertion that the triton backend doesn't support NSA with FP8 KV cache ([msg 174]). It tried flashmla_sparse as the NSA decode backend ([msg 178]), which got the server up briefly but still crashed during decode (<msg id=182-184>). Each attempt revealed a new incompatibility, a new edge case in the intersection of Blackwell hardware, SGLang's evolving codebase, and GLM-5's novel architecture.

The Hypothesis Behind This Attempt

By [msg 187], the assistant had converged on a specific hypothesis. A warning in the server logs kept appearing: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." DeepGemm was auto-selecting on Blackwell GPUs, but the GLM-5 checkpoint used a different scale format (ue8m0 vs. whatever the checkpoint actually stored). The assistant reasoned that this format mismatch was causing the GEMM (General Matrix Multiply) operations to produce garbage outputs—NaN or Inf values that propagated through the probability computation.

The fix seemed straightforward: force SGLang to use a different GEMM backend. The assistant launched a new server with --fp8-gemm-backend cutlass ([msg 187]), explicitly overriding the auto-detected DeepGemm path. This was a targeted intervention based on a clear causal chain: incompatible scale format → DeepGemm produces wrong results → NaN in logits → crash during decode.

The False Positive: Why the Server Seemed Healthy

Message [msg 195] reports that the server is up and warmup passed. This is significant because it demonstrates that the model loading and prefill phases completed without error. The server successfully:

Loaded all 83 safetensors checkpoint shards across 8 GPUs
Captured CUDA graphs for the prefill phase (as seen in [msg 179], where capture times of ~7.7 seconds were reported)
Responded to a /model_info HTTP request
Completed a prefill warmup batch (one sequence, 64 tokens) All of these operations succeeded. The assistant had every reason to be optimistic. The warmup process exercises the model's forward pass for prefill (processing input tokens in parallel), and it passed without triggering the NaN assertion. This strongly suggested that the --fp8-gemm-backend cutlass change had resolved the DeepGemm scale format issue—at least for the prefill path. But the warmup did not exercise the decode path. Decode is fundamentally different from prefill: it processes tokens one at a time, uses different CUDA kernels, and in the case of DSA models, invokes the NSA (Native Sparse Attention) decode backend. The prefill warmup only proved that the model could load and process initial inputs. It did not prove that the full generation loop was stable.

The Assumption That Failed

The assistant's implicit assumption was that fixing the DeepGemm scale format incompatibility would resolve all NaN issues. This was a reasonable hypothesis given the evidence: the warning was the only explicit error signal in the logs, and the crash symptom (NaN in probability tensor) was consistent with a GEMM-level numerical accuracy problem.

However, the assistant had also noted another warning in earlier attempts: a Transformers 5.2.0 warning about potential RoPE (Rotary Position Embedding) parameter incompatibilities. This warning was mentioned in [msg 184] but was deprioritized in favor of the more concrete DeepGemm warning. The RoPE warning was vague—it said "if you experience issues related to RoPE parameters"—and didn't provide a clear causal mechanism. The DeepGemm warning, by contrast, directly stated "accuracy degradation on Blackwell," which mapped perfectly onto the NaN symptom.

The mistake was not in the hypothesis itself but in the failure to consider that multiple issues might be contributing to the crash. The DeepGemm fix addressed one potential source of numerical instability, but the DSA attention path—which GLM-5 uniquely uses—remained unexamined. The assistant would later discover in [msg 197] that the crash persisted, leading to a new hypothesis: "The NaN is likely coming from the DSA attention path, not the MoE/GEMM path."

The Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

SGLang architecture: Understanding that SGLang separates prefill and decode into different execution paths, uses CUDA graph capture for optimization, and has multiple interchangeable backends for attention, GEMM operations, and MoE routing. The distinction between attention-backend, nsa-decode-backend, fp8-gemm-backend, and moe-runner-backend is critical.

Blackwell GPU architecture (SM120): Knowledge that Blackwell introduced new tensor core capabilities, new FP4/FP8 formats, and that SGLang needed a specific patch (PR #14311) for shared memory handling on this architecture. DeepGemm is a Blackwell-optimized GEMM library that auto-selects when available.

GLM-5 model architecture: Understanding that GLM-5 uses the glm_moe_dsa architecture, which incorporates DeepSeek Sparse Attention (DSA). DSA forces the use of NSA (Native Sparse Attention) backends in SGLang, which have their own set of compatibility constraints separate from the standard attention backends.

Quantization formats: The distinction between ue8m0 and other scale formats for FP4/FP8 quantization, and how incompatible scale formats can cause silent numerical corruption in GEMM operations.

The debugging process: The assistant had been iterating through a systematic process of elimination—changing one parameter at a time, observing the result, and forming new hypotheses. This message represents the test of one specific hypothesis (DeepGemm scale format incompatibility).

The Output Knowledge Created

This message produced a negative result—the server crashed again—but negative results are often the most informative in debugging. The failure told the assistant that:

The DeepGemm scale format was not the sole cause of the NaN issue. Even with --fp8-gemm-backend cutlass, the crash persisted.
The prefill path and decode path have different numerical stability characteristics. The warmup passed, but decode still failed.
The issue was deeper than a simple GEMM backend selection. It might involve the DSA attention kernels, the RoPE computation, or some other component unique to the decode phase. This negative result forced the assistant to broaden its search. In [msg 197], the assistant pivoted to examining the DSA attention path and the Transformers 5.2.0 RoPE warning. In [msg 198], it began investigating whether SGLang had its own model class for GLM-5 that could bypass Transformers entirely. Each of these new directions was enabled by the failure of the DeepGemm hypothesis.

The Broader Significance

Message [msg 195] exemplifies a pattern that recurs throughout complex systems debugging: the false positive. The server started successfully. The warmup passed. All the visible indicators said "green." Yet the system was fundamentally broken. This is the nightmare of deploying cutting-edge AI models on new hardware—the surface-level signals (model loaded, server running, warmup complete) are necessary but not sufficient for correct operation.

The message also illustrates the asymmetry between prefill and decode in transformer inference. Prefill is computationally intensive but structurally simple—it processes a known input in parallel. Decode is autoregressive and iterative, exposing the model to edge cases in numerical precision, kernel selection, and memory access patterns that prefill never encounters. A model that loads and prefills correctly is only half-working.

Finally, this message demonstrates the value of systematic hypothesis testing in debugging. The assistant had a clear theory (DeepGemm scale format mismatch → NaN), implemented a targeted fix (forcing cutlass backend), and tested it with a minimal query ("What is 2+2? Answer briefly."). The test was clean, the result was unambiguous, and the failure pointed clearly to the next avenue of investigation. This is debugging at its most disciplined—and its most frustrating.