The Pivot to Prior Art: Consulting Local Research to Debug a NaN Crash on Blackwell GPUs

In the middle of an intense debugging session, message [msg 191] appears deceptively simple. It contains just two file reads: the assistant reads lines 200+ from a FINDINGS.md document and the full contents of configs/kimi-k25-config.json. On its surface, this is a mundane operation — the assistant is fetching data from disk. But this message represents a critical inflection point in the conversation, a deliberate pivot from blind trial-and-error to evidence-based debugging informed by prior art. Understanding why this message exists, what assumptions drove it, and what knowledge it produced reveals the underlying structure of how complex ML deployment problems are solved in practice.

The Context: A Persistent and Mysterious Crash

To understand message [msg 191], we must first understand the crisis that preceded it. The assistant had been attempting to deploy the GLM-5-NVFP4 model — a massive mixture-of-experts (MoE) language model with DeepSeek Sparse Attention (DSA) architecture — on a machine equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each, SM120 architecture). The deployment used SGLang, a popular inference serving framework, with tensor parallelism across all 8 GPUs.

The model loaded successfully. CUDA graphs were captured. The server announced it was ready. But every time a real inference request was sent — during the decode phase, when the model actually generates tokens — the server crashed with a device-side assert triggered error. The root cause, as the assistant had just discovered in [msg 186], was that the probability tensor contained inf or nan values. The model was producing numerical garbage during generation.

The assistant had been iterating through configuration permutations for many rounds: switching attention backends (triton, flashmla_sparse, flashmla_kv), forcing different FP8 GEMM backends (cutlass), disabling CUDA graphs, and trying various NSA (Native Sparse Attention) backends. Each attempt required killing the server, relaunching with new flags, waiting 2-3 minutes for model loading, and then observing the same crash. The assistant had identified two plausible culprits: a warning about DeepGemm being enabled with an incompatible checkpoint scale format (ue8m0), and a Transformers 5.2.0 warning about potential RoPE parameter incompatibilities. But despite trying multiple fixes, the NaN crash persisted.

The User's Intervention: A Call to Consult Prior Art

At [msg 188], the user intervened with a succinct but strategically important instruction: "Read research in ./ that run other glm/kimi models." This was not a request for generic documentation. The user was pointing the assistant to a local research repository — glm-kimi-sm120-rtx6000bw/ — that contained findings from previous deployments of similar models (GLM-4.7, Kimi K2-Thinking) on the exact same hardware configuration. The implicit reasoning was: someone has already solved this class of problem on this hardware. Their findings contain the answers we need.

The assistant acknowledged this in [msg 189] and began reading. Message [msg 190] read the beginning of FINDINGS.md (lines 1-199) and listed the directory contents. Message [msg 191] — our subject — reads the continuation of FINDINGS.md and the Kimi K2-Thinking model configuration.

What Message 191 Actually Contains

The message consists of two parallel read tool calls. The first reads lines 200-208 of FINDINGS.md, which contain:

The Reasoning and Motivation: Why This Message Was Written

Message [msg 191] exists because the assistant recognized that the current trial-and-error approach was not converging. After multiple rounds of restarting the server with different flags and observing the same NaN crash, the assistant needed a new source of truth. The local research repository represented institutional knowledge — it documented what had actually worked on this exact hardware configuration.

The assistant's reasoning, visible in the subsequent message [msg 193], reveals what it was looking for:

"This is extremely valuable. The key findings for our GLM-5-NVFP4 deployment: 1. The device-side assert triggered / NaN issue is NOT the SM120 shared memory problem — the SM120 block size fix is already in our main branch install. The NaN is likely from the DeepGemm + non-ue8m0 scale format warning. 2. Previous successful NVFP4 deployment (Kimi K2-Thinking NVFP4) used: --disable-cuda-graph, --quantization modelopt_fp4, No --fp8-gemm-backend override needed. The cuDNN FP4 GEMM path worked for linear layers."

The assistant was specifically looking for:

  1. Confirmation that the NaN issue was not the SM120 shared memory bug — the FINDINGS.md documented that the SM120 fix was already incorporated, ruling out one hypothesis.
  2. The exact flags used in a successful NVFP4 deployment — the Kimi K2-Thinking deployment had worked, and its configuration could serve as a template.
  3. Whether the DeepGemm scale format issue was known — the FINDINGS.md would confirm if this was a documented problem.

Assumptions Made

Several assumptions underpin this message:

Assumption 1: Prior art is transferable. The assistant assumes that findings from deploying Kimi K2-Thinking NVFP4 and GLM-4.7 on the same hardware will generalize to GLM-5-NVFP4. This is reasonable — the hardware is identical, the quantization method is the same (NVFP4), and the serving framework is the same (SGLang). However, GLM-5 uses a different architecture (glm_moe_dsa with DSA attention) which introduces new variables.

Assumption 2: The research repository is authoritative. The assistant trusts that the FINDINGS.md was written by someone who thoroughly debugged these issues and documented accurate solutions. In practice, research notes can contain errors, omissions, or stale information.

Assumption 3: The NaN crash has a single root cause. The assistant is looking for one configuration change that will fix everything. In reality, the crash could be caused by multiple interacting factors — the DeepGemm scale format issue, the Transformers 5.2.0 RoPE incompatibility, and the DSA attention backend all potentially contributing.

Assumption 4: The configuration flags are the primary lever. The assistant assumes that the solution lies in changing SGLang server arguments. It does not consider other dimensions like patching the model code, modifying the checkpoint, or using a different inference framework entirely.

Input Knowledge Required

To understand message [msg 191], the reader needs:

  1. Knowledge of the GLM-5-NVFP4 model: It's a quantized (NVFP4) version of GLM-5, a large MoE language model with DeepSeek Sparse Attention (DSA). The NVFP4 quantization uses 4-bit floating point for weights, which is challenging to deploy correctly.
  2. Knowledge of Blackwell GPU architecture (SM120): The RTX PRO 6000 Blackwell uses the new SM120 architecture, which has different shared memory characteristics and kernel compatibility requirements compared to Hopper (SM90) or Ada Lovelace (SM89).
  3. Knowledge of SGLang serving framework: Understanding what --attention-backend, --fp8-gemm-backend, --moe-runner-backend, --kv-cache-dtype, and --nsa-decode-backend flags control is essential.
  4. Knowledge of DeepGemm: DeepGemm is a specialized GEMM (General Matrix Multiply) library optimized for Blackwell. It uses a specific scale format (ue8m0) that may not be compatible with all checkpoints.
  5. Knowledge of the conversation history: The reader needs to know that the assistant has been debugging a NaN crash for many rounds, that the server has been killed and restarted multiple times, and that the user explicitly asked the assistant to consult the local research.

Output Knowledge Created

Message [msg 191] produces several pieces of knowledge:

  1. Memory budget confirmation: The FINDINGS.md reveals that a similar model (GLM-4.7) uses ~85GB per GPU with ~12.48GB available after load. This confirms the memory budget is tight but feasible for GLM-5-NVFP4.
  2. Quantization method details: The FINDINGS.md documents that the quantization method is CompressedTensorsW8A8Fp8MoEMethod with BF16 activations and FP8 weights — a different approach than the modelopt_fp4 quantization used in the current deployment.
  3. Kimi K2-Thinking architecture reference: The kimi-k25-config.json provides a reference architecture configuration that can be compared with GLM-5's configuration to identify differences that might cause the NaN issue.
  4. Confirmation of the DeepGemm issue's existence: The FINDINGS.md documents the DeepGemm scale format problem as a known issue on Blackwell, validating the assistant's hypothesis from [msg 186].
  5. A template for the next attempt: The assistant learns that the previous successful NVFP4 deployment used --disable-cuda-graph and did not need an explicit --fp8-gemm-backend override, suggesting the cuDNN path worked. This directly informs the next server launch configuration.

Mistakes and Incorrect Assumptions

The most significant mistake visible in this message is the assumption that the DeepGemm scale format issue is the primary cause of the NaN crash. While the FINDINGS.md confirms the issue exists, the assistant would later discover (in subsequent messages beyond this chunk) that the NaN crash persisted even after forcing --fp8-gemm-backend cutlass. The DeepGemm warning was a red herring — or at least not the sole cause.

A second issue is the over-reliance on configuration flags as the debugging mechanism. The assistant had been iterating through combinations of --attention-backend, --nsa-decode-backend, --fp8-gemm-backend, and --kv-cache-dtype flags for many rounds. Each iteration required 2-3 minutes of server startup time. The local research repository suggested that --disable-cuda-graph was important, but the assistant had not yet tried this flag in combination with the correct attention backend.

A third subtle issue is that the assistant reads only specific portions of FINDINGS.md (lines 200+). It had read lines 1-199 in [msg 190]. But the file is long (over 500 lines), and the assistant may be missing critical information in the middle sections — such as specific error messages, workarounds, or configuration nuances that are directly relevant to the GLM-5 deployment.

The Thinking Process Visible in the Message

While message [msg 191] itself contains no explicit reasoning text — it is purely a tool call output — the thinking process is visible in its structure and timing. The assistant chose to read two specific files in parallel: the continuation of FINDINGS.md and the Kimi K2-Thinking config. This reveals a deliberate strategy:

  1. Read the findings document to understand what problems were encountered and solved on this hardware.
  2. Read a successful model configuration to use as a reference template. The parallel execution (both reads happen simultaneously) shows that the assistant considers these independent pieces of information that can be fetched concurrently. The assistant is not waiting to see what FINDINGS.md says before deciding to read the config — it already knows both will be valuable. The subsequent message ([msg 193]) confirms this thinking. The assistant immediately synthesizes the findings into actionable insights: the NaN issue is not the SM120 bug, the DeepGemm warning is the likely culprit, and the previous successful deployment used --disable-cuda-graph. This synthesis is the real output of message [msg 191] — the raw data is just the input to the reasoning process.

Conclusion

Message [msg 191] is a turning point in the debugging session. It represents the shift from random exploration to informed investigation, from guessing configuration flags to studying documented solutions. The assistant recognized that the trial-and-error approach was not converging and that the local research repository contained institutional knowledge that could shortcut the debugging process. While the DeepGemm hypothesis would ultimately prove insufficient to fully resolve the NaN crash, the act of consulting prior art was strategically sound. In complex ML deployment scenarios where the interaction surface between model architecture, GPU architecture, and serving framework creates emergent bugs, the most valuable resource is often not documentation or source code — it's the hard-won knowledge of what actually worked on the same hardware before.