The Diagnostic Pivot: How Local Research Unraveled a NaN Mystery on Blackwell GPUs

Introduction

In the high-stakes world of deploying cutting-edge large language models on novel hardware architectures, the difference between a successful deployment and a silent failure often comes down to a single insight. Message 193 in this opencode session captures exactly such a moment — a diagnostic pivot where the assistant, after hours of iterative debugging against a persistent NaN crash during decode, finally synthesizes information from a local research repository to formulate a coherent theory of the root cause. This message is the turning point in a complex debugging saga: the deployment of GLM-5-NVFP4, a quantized mixture-of-experts (MoE) model with sparse attention, across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using the sglang serving framework.

The message is deceptively brief — a few paragraphs of analysis followed by a bash command to check server status. But within those paragraphs lies a dense web of reasoning that connects observations from multiple failed attempts, cross-references a previously successful deployment of a similar model family, and identifies a critical architectural difference that may explain the persistent failure. This article unpacks that reasoning in detail.

The Context: A Debugging Session at an Impasse

To understand message 193, one must first appreciate the debugging landscape that preceded it. The session had been attempting to deploy GLM-5-NVFP4 — a 4-bit quantized version of the GLM-5 model using NVIDIA's ModelOpt FP4 quantization — on a machine equipped with 8 RTX PRO 6000 Blackwell GPUs (96 GB each, SM120 architecture). The deployment used sglang, built from the main branch to include a critical SM120 shared memory fix (PR #14311).

The pattern was maddeningly consistent: the server would load successfully, capture CUDA graphs during prefill warmup, and then crash during decode with a device-side assert triggered error. The root cause, as revealed in the logs, was that the "probability tensor contains either inf, nan or element < 0" — a numerical stability failure in the model's output distribution.

Multiple configuration attempts had failed to resolve this. The assistant had tried switching attention backends (triton, flashmla_sparse), forcing --fp8-gemm-backend cutlass, disabling CUDA graphs, and experimenting with various NSA (Native Sparse Attention) backends. Each attempt ended with the same NaN crash. The team was at an impasse, cycling through configuration permutations without a clear theory of the root cause.

Then the user issued a simple but pivotal instruction: "Read research in ./ that run other glm/kimi models" ([msg 188]). This directive sent the assistant to examine the local research repository — a collection of findings, configurations, and patches from previous deployments of similar models on the same hardware. Message 193 is the assistant's synthesis of what it found there.

The Message: A Three-Part Diagnostic Synthesis

The message opens with a bold declarative statement: "This is extremely valuable." This is not casual enthusiasm — it signals a genuine breakthrough in understanding. The assistant then organizes its findings into three numbered points, each connecting a previously observed symptom to a potential root cause.

Point 1 addresses the most persistent red herring in the debugging session: the SM120 shared memory issue. Throughout the earlier attempts, the assistant had been operating under the assumption that the crash might be related to Blackwell's new SM120 architecture requiring special handling for CUDA block sizes. The sglang main branch build was specifically chosen to include PR #14311, which addressed exactly this issue. But after reading the findings, the assistant now argues that this is not the problem — the SM120 fix is already in place. Instead, the NaN is "likely from the DeepGemm + non-ue8m0 scale format warning." This is a critical reframing: the issue is not a kernel compatibility problem but a numerical accuracy problem arising from incompatible quantization scale formats.

Point 2 introduces the most powerful piece of evidence: a previous successful deployment of Kimi K2-Thinking NVFP4 on the exact same hardware. This earlier deployment used specific flags — --disable-cuda-graph, --quantization modelopt_fp4, no --fp8-gemm-backend override — and crucially, "the cuDNN FP4 GEMM path worked for linear layers." This establishes a precedent: the hardware can run NVFP4 models successfully. The failure is not inherent to the GPU architecture or the quantization scheme.

Point 3 identifies the key architectural difference that explains why GLM-5 fails where Kimi K2 succeeds: "GLM-5 uses glm_moe_dsa architecture with DSA (DeepSeek Sparse Attention) — this forces NSA backends (flashmla_kv) which may be the source of the NaN." This is the insight that ties everything together. The sparse attention mechanism, unique to GLM-5, forces the use of NSA attention backends that may interact badly with the FP4 quantization or the Blackwell architecture.

The Reasoning Process: Connecting Disparate Observations

What makes this message remarkable is the reasoning structure. The assistant is not simply reporting what it read — it is actively synthesizing across multiple information sources:

  1. Log output from failed attempts: The device-side assert error, the DeepGemm scale format warning, the transformers 5.2.0 RoPE warning
  2. The FINDINGS.md document: Details of the Kimi K2-Thinking NVFP4 deployment, including successful flags and configurations
  3. The model architecture: Understanding that GLM-5 uses glm_moe_dsa (a DeepSeek-style architecture with sparse attention) while Kimi K2 uses standard attention
  4. The sglang codebase: Knowledge that DSA architectures automatically force NSA backend selection, overriding the general attention backend setting The assistant is effectively building a differential diagnosis: "Kimi K2 works on this hardware with NVFP4; GLM-5 does not. What is different?" The answer it arrives at is the DSA/NSA attention mechanism. This is a classic scientific reasoning pattern — control for variables, identify the difference, and hypothesize that the difference causes the observed effect.

Assumptions and Potential Pitfalls

The message makes several assumptions that deserve scrutiny. First, it assumes that the Kimi K2 deployment is a valid control — that the hardware, software stack, and quantization method are sufficiently similar to isolate the DSA attention as the differentiating factor. While the FINDINGS.md confirms NVFP4 quantization on the same 8x RTX PRO 6000 Blackwell setup, there could be other differences: different model size, different MoE configuration, different tokenizer behavior, or different sglang version at the time of deployment.

Second, the assistant assumes that the DeepGemm scale format warning is the primary cause of the NaN, rather than, say, the RoPE parameter incompatibility warned about by transformers 5.2.0. The warning about DeepGemm being enabled with a non-ue8m0 checkpoint scale format is indeed a strong candidate — it directly addresses numerical accuracy — but the transformers warning about "issues related to RoPE parameters" could also produce NaN logits if rotary position embeddings are computed incorrectly. The assistant implicitly prioritizes the DeepGemm hypothesis, perhaps because the RoPE warning is more generic while the DeepGemm warning is specific to Blackwell and NVFP4.

Third, the message assumes that disabling --kv-cache-dtype fp8_e4m3 (which it proposes in the action plan) will help by avoiding the flashmla KV cache path. This is a reasonable inference — if the NSA backends are the source of the NaN, and FP8 KV cache forces NSA backend selection, then removing FP8 KV cache should bypass the problematic code path. But this assumption may be incorrect if the NaN originates elsewhere in the model's forward pass.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning multiple domains:

Output Knowledge Created

The message produces several valuable outputs:

  1. A reframed diagnostic hypothesis: The NaN crash is not an SM120 kernel compatibility issue but a numerical accuracy problem caused by DeepGemm scale format incompatibility, potentially amplified by NSA attention backends
  2. A concrete action plan: Try without --kv-cache-dtype fp8_e4m3 and with --disable-cuda-graph, mirroring the successful Kimi K2 configuration
  3. A documented precedent: The Kimi K2-Thinking NVFP4 deployment on the same hardware validates that the hardware and quantization method are viable
  4. A reusable reasoning pattern: The method of comparing a failing deployment against a successful one on identical hardware to isolate architectural differences

The Bash Command as Epilogue

The message ends with a bash command checking the server log, showing the model still loading shards (14-25% progress). This is a deliberate choice — the assistant is not waiting for the server to finish before reporting its findings. It is capturing the reasoning in the moment, while the new attempt is still loading. This gives the article a sense of real-time debugging, where analysis and action interleave. The command also serves a practical purpose: it confirms that the previous attempt (with --fp8-gemm-backend cutlass) is still loading, meaning the assistant hasn't yet seen whether that configuration works. The new plan (disabling FP8 KV cache) is formulated before the previous attempt finishes, based purely on the research findings.

Conclusion

Message 193 represents the moment when a debugging session transitions from trial-and-error to hypothesis-driven investigation. By reading the local research repository, the assistant gains access to a crucial piece of evidence — a successful deployment of a similar model on identical hardware — that allows it to reframe the problem. The NaN crash is not a generic SM120 incompatibility but a specific interaction between GLM-5's DSA sparse attention, the NSA backends it forces, and the DeepGemm scale format mismatch. This insight would not have been possible without the local findings, which is why the user's instruction to "read research in ./" was so consequential.

The message also illustrates a broader principle of AI-assisted debugging: the most powerful diagnostic tool is not exhaustive parameter search but the ability to synthesize information across disparate sources — log output, codebase knowledge, model architecture, and historical precedent — into a coherent causal theory. Message 193 is a case study in exactly that kind of synthesis.