The Moment of Insight: Tracing NaN Logits to a DeepGemm Scale Format Mismatch on Blackwell

In the high-stakes world of deploying cutting-edge large language models on novel GPU architectures, the difference between a functioning server and a silent crash often comes down to a single, overlooked warning. Message 186 in this opencode session represents a pivotal diagnostic breakthrough — the moment when the assistant, after a long chain of failed experiments and misdirected hypotheses, finally identifies the true root cause of a persistent crash during model inference. This message is a masterclass in systematic debugging, showing how careful attention to error messages and log warnings can cut through hours of fruitless trial and error.

The Context: A Server That Won't Survive Decode

To understand the significance of message 186, we must first appreciate the struggle that preceded it. The assistant had been tasked with deploying the GLM-5-NVFP4 model — a massive Mixture-of-Experts (MoE) language model quantized to FP4 — across eight NVIDIA RTX PRO 6000 Blackwell GPUs using the SGLang inference framework. This was already a challenging proposition: Blackwell (SM120) is a brand-new GPU architecture, the model uses an unusual glm_moe_dsa architecture with sparse attention, and the quantization scheme (modelopt_fp4) was itself cutting-edge.

The server had successfully loaded the model, captured CUDA graphs for various batch sizes, and passed its initial warmup prefill. But every single time a real inference request came in, the server crashed during the decode phase with a device-side assert triggered CUDA error. The assistant had spent several rounds chasing what seemed like the obvious culprit: the attention backend. It tried --attention-backend triton, which crashed with an assertion about NSA and FP8 KV cache incompatibility. It tried --nsa-decode-backend flashmla_sparse, which also crashed. It tried disabling CUDA graphs, switching to trtllm backends — all with the same result. Each attempt was a reasonable hypothesis, and each one failed.

The Message: A Diagnostic Pivot

Message 186 begins with the assistant quoting the critical error that had been hiding in the logs:

"probability tensor contains either inf, nan or element < 0" — this is the key. The model is producing NaN/Inf logits during decode. This is NOT an SM120 kernel issue — it's a numerical accuracy problem.

This single sentence represents a fundamental shift in the assistant's mental model of the bug. Up until this point, the working assumption had been that the crash was a kernel compatibility issue — that the CUDA kernels for attention or MoE computation simply didn't work correctly on the SM120 architecture. This was a reasonable assumption: Blackwell is new, and many CUDA kernels need architecture-specific tuning. The assistant had even consulted GitHub issues about SM120 problems and tried multiple attention backends designed for different architectures.

But the error message itself told a different story. A device-side assert triggered error can have many causes, but the specific assertion — that the probability tensor contains NaN, Inf, or negative values — points squarely at numerical instability. The model's forward pass is producing garbage values in its output logits, and when the sampling kernel tries to draw from this corrupted probability distribution, it triggers the assertion.

Connecting the Dots: The DeepGemm Warning

The assistant's next move is the crucial one. It recalls a warning that appeared in the server logs during earlier runs but had been overshadowed by the more dramatic crash errors:

The warning about "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell" is likely the culprit.

The assistant then executes a targeted grep command to confirm this warning's presence:

[2026-02-19 00:09:54] WARNING model_config.py:955: DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
[2026-02-19 00:09:55] DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
[2026-02-19 00:10:02 TP2] DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
[2026-02-19 00:10:02 TP6] DeepGemm is enabled b...

The warning repeats across multiple tensor-parallel ranks (TP2, TP6), confirming it's a systemic issue, not a one-off log line. This is the moment of convergence: two independent pieces of evidence — the NaN probability tensor and the DeepGemm scale format warning — point to the same root cause.

Why DeepGemm Matters

DeepGemm is a high-performance kernel library for mixed-precision matrix multiplication, particularly optimized for FP4 and FP8 computations on NVIDIA GPUs. When SGLang loads a quantized model, it selects a "gemm backend" (general matrix multiply) to handle the actual computation. On Blackwell GPUs, the auto setting selects DeepGemm because it offers the best performance for the architecture.

However, DeepGemm expects a specific scale factor format called ue8m0 for its internal quantization. The GLM-5-NVFP4 checkpoint was quantized with a different scale format — likely the format used by the model's original quantization tool (NVIDIA's ModelOpt). When DeepGemm interprets the scale factors in the wrong format, it produces incorrect results: the matrix multiplications yield values that are off by orders of magnitude, cascading into NaN or Inf logits.

The warning phrase "accuracy degradation" is dangerously understated. In practice, "accuracy degradation" for a language model means the output logits become numerical garbage — exactly the NaN/Inf values that crash the server.

The Assumptions That Led Astray

Message 186 is also valuable for what it reveals about the debugging process's implicit assumptions. The assistant had been operating under several hypotheses:

  1. The SM120 kernel hypothesis: That the crash was caused by CUDA kernels lacking proper SM120 support. This led to trying different attention backends (triton, flashmla_sparse, flashmla_kv) and NSA backends (trtllm).
  2. The Transformers RoPE hypothesis: That Transformers 5.2.0 had a RoPE parameter incompatibility causing out-of-bounds accesses. This was explicitly suggested by a warning in the logs.
  3. The KV cache dtype hypothesis: That FP8 KV cache interacted badly with certain attention backends. Each of these was plausible, and each led to several rounds of experimentation. The assistant's earlier message (184) explicitly states: "This is likely the RoPE issue with transformers 5.2.0 that the warning mentioned." This was a reasonable inference — RoPE miscalculations can indeed cause out-of-bounds indices that trigger device-side assertions. But message 186 represents a critical re-evaluation. By looking past the generic "device-side assert" error and examining the specific assertion message — "probability tensor contains either inf, nan or element < 0" — the assistant realizes the crash is not about out-of-bounds memory access but about numerical corruption. This is a fundamentally different class of bug.

The Input Knowledge Required

To understand message 186 fully, one needs knowledge spanning several domains:

The Output Knowledge Created

Message 186 produces several critical insights that shape the subsequent debugging:

  1. The crash is numerical, not architectural: The root cause is not missing SM120 kernel support but a quantization scale format mismatch. This completely changes the solution space.
  2. The fix is a configuration change, not a code change: Rather than patching kernels or waiting for SGLang updates, the solution is to force a different gemm backend. The assistant immediately follows up (in message 187) with a launch command that sets --fp8-gemm-backend cutlass.
  3. The warning was the real signal: The DeepGemm scale format warning, which might have been dismissed as a minor accuracy concern, was actually the critical diagnostic clue. This reinforces the principle that all warnings deserve attention, especially on new hardware.
  4. The model card's recommended parameters matter: The HuggingFace model card for GLM-5-NVFP4 specifies --moe-runner-backend flashinfer_cutlass but does not explicitly mention --fp8-gemm-backend. The assistant realizes that the model card's recommendations implicitly avoid DeepGemm by specifying a different runner backend.

The Broader Significance

Message 186 is more than just a debugging step — it's a case study in how to systematically diagnose failures at the intersection of new hardware, new models, and new serving frameworks. The assistant demonstrates several best practices:

Conclusion

Message 186 is the turning point in a complex debugging session. By recognizing that a "device-side assert" error was actually a numerical accuracy problem caused by a DeepGemm scale format mismatch, the assistant saves what could have been hours or days of further kernel debugging. The message demonstrates that the most valuable debugging skill is not exhaustive trial-and-error but the ability to read error messages carefully, connect disparate pieces of evidence, and revise one's mental model of the problem. In the world of deploying frontier AI models on next-generation hardware, that skill is worth its weight in GPU memory.