The Moment of Cautious Optimism: Debugging NaN Crashes on Blackwell GPUs

Introduction

In the complex world of deploying large language models on cutting-edge hardware, the gap between a model loading successfully and actually generating coherent text can be vast. Message 194 in this opencode session captures one such pivotal moment — a checkpoint where an AI assistant, after rounds of iterative debugging, believes it has identified the root cause of persistent NaN crashes during model decode, and waits with cautious optimism to see if its hypothesis holds. This message, nestled between intensive research and the eventual revelation of yet another failed attempt, offers a fascinating window into the reasoning process of an AI system debugging a production ML deployment.

The Context: A Long Debugging Journey

To understand message 194, we must first appreciate the journey that led to it. The assistant had been attempting to deploy the GLM-5-NVFP4 model — a large Mixture-of-Experts (MoE) language model with DeepSeek Sparse Attention (DSA) — across 8 NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture, 96 GB each) using the SGLang inference server. The deployment had been plagued by a persistent crash: a device-side assert triggered error caused by NaN/Inf values in the probability tensor during the decode phase. This crash occurred reliably every time a generation request was made, despite the model loading successfully and the CUDA graph capture completing without incident.

The assistant had already attempted multiple configurations. It tried the flashinfer attention backend ([msg 170]), which crashed. It tried the triton attention backend ([msg 174]), which produced an assertion error because the triton backend doesn't support NSA (Native Sparse Attention) with FP8 KV cache. It tried flashmla_sparse as the NSA decode backend ([msg 178]), which also crashed with the same NaN error. Each attempt brought the assistant closer to understanding the problem, but none had succeeded.

The breakthrough came in message 186, when the assistant traced the crash to a specific warning in the server logs: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." DeepGemm, a high-performance GEMM (General Matrix Multiply) library, was auto-selecting on the Blackwell GPUs but encountering an incompatible checkpoint scale format, potentially producing NaN outputs. The assistant launched a new server in message 187 with --fp8-gemm-backend cutlass to force the system away from DeepGemm.

Then the user intervened in message 188, asking the assistant to read research files from a local repository (FINDINGS.md) that documented previous deployments of similar models on the same hardware. The assistant spent messages 190–193 reading and analyzing these findings, which confirmed that Kimi K2-Thinking NVFP4 had deployed successfully on the same hardware configuration.

The Message Itself

Message 194 is the assistant's response after synthesizing the research findings:

Still loading with the --fp8-gemm-backend cutlass config. Let me let it finish and test, since the DeepGemm warning was the likely NaN source. Let me wait:

>

`` [bash] sleep 180 && ssh 10.1.230.175 'tail -10 ~/sglang-glm5.log' [2026-02-19 00:17:22 TP2] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [2026-02-19 00:17:24 TP0] max_total_num_tokens=425664, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=64, context_len=202752, available_gpu_mem=6.60 GB [2026-02-19 00:17:24] INFO: Started server process [18476] [2026-02-19 00:17:24] INFO: Waiting for application startup. [2026-02-19 00:17:24] INFO: ... ``

The message is deceptively simple: a brief statement of intent, a bash command to wait and check logs, and the resulting log output. But beneath this surface lies a rich tapestry of reasoning, decision-making, and assumptions.

The Reasoning Process: Hypothesis Formation and Testing

The assistant's reasoning in this message is explicit and methodical. The phrase "since the DeepGemm warning was the likely NaN source" reveals a causal hypothesis that has been formed through the iterative debugging process. The assistant has connected the dots between:

  1. The warning about DeepGemm and incompatible scale format (ue8m0 vs the checkpoint's format)
  2. The NaN/Inf values appearing in the probability tensor during decode
  3. The fact that DeepGemm auto-selects on Blackwell (SM120) GPUs
  4. The successful deployment of Kimi K2-Thinking NVFP4, which presumably didn't trigger this warning The decision to "let it finish and test" rather than kill the server and restart with yet another configuration shows a deliberate choice to follow through on the current hypothesis before pivoting. This is a hallmark of methodical debugging: change one variable at a time, observe the result, and only then formulate the next experiment. Notably, the assistant had already launched this server in message 187, before the user asked it to read the research files. The research reading (messages 190–193) took time — the server was loading in the background during those exchanges. Message 194 represents the assistant returning to check on that background process, armed with new knowledge from the research repository that reinforced its hypothesis.

Assumptions and Their Validity

Message 194 rests on several key assumptions, some of which would prove incorrect:

Assumption 1: The DeepGemm scale format incompatibility is the root cause of the NaN. This was the central hypothesis. The assistant had strong evidence: a specific warning about accuracy degradation on Blackwell, combined with the NaN crash occurring only during decode (when GEMM operations are most active). However, as revealed in subsequent messages ([msg 197]), this assumption was incorrect. The server crashed with the same NaN error even with --fp8-gemm-backend cutlass, forcing the assistant to reconsider and eventually trace the issue to the DSA attention path instead.

Assumption 2: --fp8-gemm-backend cutlass would successfully override the auto-selected DeepGemm backend. The assistant assumed that explicitly specifying the cutlass backend would prevent DeepGemm from being used. While this flag should theoretically work, the persistence of the NaN suggested either that the flag wasn't being honored, that the FP4 GEMM path (separate from FP8) was still using DeepGemm, or that the NaN originated from a different part of the model entirely.

Assumption 3: The server process (PID 18476) was still running and would complete loading. The log output confirmed this assumption — the server was progressing through shard loading and had started its process. The available GPU memory of 6.60 GB per GPU and context length of 202,752 tokens indicated successful model loading.

Assumption 4: The research findings supported the DeepGemm hypothesis. The assistant synthesized the FINDINGS.md content in message 193, noting that the previous successful Kimi K2-Thinking NVFP4 deployment used no --fp8-gemm-backend override. The assistant interpreted this as evidence that the cuDNN FP4 GEMM path worked for that model, but failed to account for the critical architectural difference: GLM-5 uses DSA (DeepSeek Sparse Attention), which the Kimi model did not.

Input Knowledge Required

To fully understand message 194, one needs knowledge of several technical domains:

Output Knowledge Created

The log output in message 194 provides several valuable pieces of information:

  1. The server is loading successfully — shards are being processed, the server process has started (PID 18476), and it's waiting for application startup
  2. Memory utilization — after model loading, each GPU has 6.60 GB of available memory, indicating the model fits within the 96 GB per GPU budget
  3. Configuration parameters — the server is using chunked prefill size of 8192, max prefill tokens of 16384, and a context length of 202,752 tokens
  4. The server is not immediately crashing — unlike some previous attempts that crashed during model loading, this one reached the "Waiting for application startup" phase This output represents a checkpoint in the debugging process. The assistant now knows the server has loaded and is ready for testing. The next step would be to send a generation request and observe whether the NaN crash still occurs.

The Broader Significance

Message 194 is significant because it captures a universal moment in debugging: the point where a developer (or in this case, an AI assistant) believes they have found the root cause and waits to confirm. The log output showing successful loading creates a sense of anticipation — will this be the fix?

The message also illustrates the value of persistence and methodical testing. The assistant had tried multiple configurations, each eliminating one possible cause. The DeepGemm hypothesis was reasonable and well-supported by the available evidence. Even though it ultimately proved incorrect, the process of testing it was necessary to narrow down the actual cause.

Furthermore, this message demonstrates the importance of domain knowledge in debugging. The assistant needed to understand GPU architectures, quantization formats, attention mechanisms, and inference server configuration to even formulate the hypothesis. The research repository (FINDINGS.md) provided crucial context about what had worked on this hardware before.

Conclusion

Message 194 captures a moment of cautious optimism in a complex debugging journey. The assistant, armed with a well-reasoned hypothesis about DeepGemm scale format incompatibility, waits to see if forcing the cutlass backend will resolve the persistent NaN crashes. The log output shows a server that has loaded successfully — a good sign, but not yet proof of success. The subsequent messages would reveal that this hypothesis was incorrect, leading the assistant to trace the NaN to the DSA attention path instead. But the methodical approach — form a hypothesis, test it, observe the result, and iterate — is precisely what makes effective debugging possible. In the high-stakes world of deploying cutting-edge AI models on new GPU architectures, such systematic reasoning is not just helpful but essential.