The NaN Revelation: How a Single Grep Command Reframed a Debugging Odyssey

In the midst of a grueling debugging session deploying the GLM-5-NVFP4 model across eight Blackwell RTX PRO 6000 GPUs, a single message — message index 185 — stands as the turning point where speculation gave way to evidence. The message is deceptively simple: a bash command that greps a server log for the phrase "Scheduler hit an exception," returning the three lines that precede it. But the output it retrieves fundamentally reshapes the entire investigation, redirecting the assistant's focus from a hardware compatibility hypothesis to a numerical accuracy problem.

The Message in Full

The assistant executes:

ssh 10.1.230.175 'grep -B3 "Scheduler hit an exception" ~/sglang-glm5.log | head -15'

And the log reveals:

] Assertion `probability tensor contains either `inf`, `nan` or element < 0] Assertion `probability tensor contains either `inf`, `nan` or element < 0: _assert_async_cuda_kernel` failed.
: block: [0` failed.
,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
[2026-02-19 00:13:33 TP5] Scheduler hit an exception: Traceback (most recent call last):
--
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

These fragmented, multi-line error messages — garbled by concurrent GPU thread logging — contain the single most important clue in the entire debugging session: the probability tensor produced by the model contains NaN (Not a Number) or Inf (Infinity) values, or negative elements. This is not a kernel crash from an unsupported instruction set. This is a numerical stability failure.

The Context That Led Here

To understand why this message is so significant, one must appreciate the debugging path that preceded it. The assistant had been engaged in a multi-hour effort to deploy GLM-5-NVFP4, a massive Mixture-of-Experts model quantized to FP4 precision, on a machine with eight Blackwell GPUs. The deployment used SGLang, a high-performance inference serving framework, built from the main branch to include a critical SM120 shared memory fix (PR #14311).

The initial server launch succeeded: weights loaded across all eight GPUs (63 GB each), KV cache was allocated (425,664 tokens, 24.37 GB), CUDA graphs were captured for batch sizes ranging from 1 to 64, and the server reported itself as running. But the very first inference request caused an immediate crash. The error was a device-side assert triggered — a CUDA runtime assertion failure that occurs when a GPU kernel detects an illegal state.

For the next several messages (messages 163 through 184), the assistant pursued a hypothesis that the crash was caused by attention backend incompatibility with the SM120 Blackwell architecture. This was a reasonable assumption. Blackwell (compute capability SM120) is a new GPU architecture, and SGLang's attention kernels — particularly the sparse NSA (Native Sparse Attention) backends used by the GLM-5 model's glm_moe_dsa architecture — may not have full SM120 support. The assistant tried:

The Turning Point

Message 185 breaks this cycle of speculation. Instead of trying yet another backend configuration and waiting for the server to crash, the assistant takes a step back and performs forensic analysis on the crash log. The grep -B3 &#34;Scheduler hit an exception&#34; command retrieves the three lines immediately preceding the scheduler exception — the lines that contain the actual CUDA kernel assertion message.

What those lines reveal is unambiguous: the probability tensor — the output of the model's final layer that gets passed to the sampling algorithm — contains NaN or Inf values. This is a numerical accuracy problem, not a kernel compatibility problem. The CUDA kernels are executing correctly; they are producing mathematically invalid results.

The Assumptions and Mistakes Exposed

This message exposes several assumptions that had been guiding the investigation:

Assumption 1: The crash is an SM120 kernel compatibility issue. The assistant had been operating under the belief that the device-side assert triggered error was caused by attention kernels that don't support the Blackwell architecture. The NaN error reveals that the kernels do run — they just produce garbage. This is a fundamentally different class of problem.

Assumption 2: The attention backend choice determines stability. The assistant had tried flashinfer, triton, and flashmla_sparse backends, assuming the crash was in the attention computation. The NaN in the probability tensor could originate from anywhere in the model — the MoE routing, the quantization/dequantization, the linear layers, or the attention mechanism itself.

Assumption 3: The warmup success indicates a working configuration. The flashmla_sparse attempt (message 178) completed warmup successfully — model loading, KV cache allocation, CUDA graph capture, and even a prefill batch all completed without error. The crash only occurred during the first real decode step. This pattern (prefill works, decode fails) is consistent with numerical instability that accumulates during the autoregressive generation process.

The message also reveals a subtle mistake in the assistant's debugging methodology: it had been looking at the type of error (device-side assert triggered) rather than the content of the assertion message. The CUDA runtime reports "device-side assert triggered" for any kernel-level assertion failure, whether it's an out-of-bounds memory access, a misaligned address, or a NaN check. The assistant had been treating all these as equivalent, when the specific assertion message contains the diagnostic key.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of CUDA error reporting. The device-side assert triggered error is a CUDA runtime mechanism where GPU kernels can call assert() to validate conditions. The assertion message is written to a special buffer and reported at the next CUDA API call. The garbled formatting (with bracket fragments like : block: [0 failed.`) is characteristic of concurrent GPU threads writing to the same error buffer.
  2. Understanding of SGLang's architecture. SGLang runs multiple tensor parallelism (TP) processes, one per GPU. The [2026-02-19 00:13:33 TP5] prefix indicates the error originated on TP rank 5 (the sixth GPU). The "Scheduler hit an exception" is SGLang's catch-all for unrecoverable errors during batch processing.
  3. Familiarity with the GLM-5-NVFP4 model. This is a quantized version of the GLM-5 model using NVIDIA's ModelOpt FP4 quantization. FP4 quantization is extremely aggressive (4-bit floating point), and numerical stability is a known challenge — especially on new GPU architectures where the quantization kernels may not be fully optimized.
  4. Awareness of the DeepGemm library. DeepGemm is a specialized GEMM (General Matrix Multiply) library that SGLang auto-selects on Blackwell GPUs. The assistant later discovers (in message 186) that DeepGemm was emitting a warning about the checkpoint's scale format (ue8m0) being incompatible, which directly causes the NaN outputs.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. The crash is a numerical stability failure, not a kernel compatibility failure. This reframes the entire debugging effort. The assistant can stop trying different attention backends and instead focus on what causes NaN/Inf in the probability tensor.
  2. The error occurs during decode, not prefill. The fact that the prefill batch (message 180) completed successfully while the decode step crashes suggests the numerical instability is specific to the autoregressive generation path — possibly related to how the KV cache interacts with the quantized weights during incremental decoding.
  3. The error is reproducible across multiple attention backends. Since both flashinfer (original attempt) and flashmla_sparse (message 178 attempt) produce the same NaN error, the root cause is independent of the attention implementation.
  4. The error manifests in the probability tensor, which is the output of the model's final linear layer (lm_head) after softmax. The assertion checks that probabilities are non-negative, finite, and sum to approximately 1. NaN or Inf values here mean the model's internal computations have diverged.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is a masterclass in forensic debugging. Rather than continuing the cycle of "change configuration, restart, wait, observe crash," the assistant pauses to analyze the crash artifact itself. The choice of grep -B3 (show 3 lines before the match) is deliberate — the assistant knows that the scheduler exception is a catch-all handler and the real error message appears in the log output just before it.

The assistant also uses head -15 to limit output, showing awareness that the log might contain many scheduler exceptions (from multiple TP ranks crashing in sequence) and only the first few are needed for diagnosis.

The fragmented nature of the output — with bracket fragments, line breaks in the middle of assertion messages, and the telltale : block: [0 failed.` — is itself informative. The assistant can infer that multiple GPU threads are writing to the error buffer simultaneously, creating interleaved output. This confirms the crash is happening in a CUDA kernel that is executing across multiple threads and blocks.

The Aftermath

The knowledge gained from message 185 directly drives the next phase of debugging. In message 186, the assistant immediately recognizes the significance: "'probability tensor contains either inf, nan or element < 0' — this is the key. The model is producing NaN/Inf logits during decode. This is NOT an SM120 kernel issue — it's a numerical accuracy problem." The assistant then checks the DeepGemm warning about scale format incompatibility, which had been present in the logs since the first server launch but was previously overlooked.

This leads to the correct fix: forcing --fp8-gemm-backend cutlass to disable DeepGemm, which resolves the NaN issue because the CUTLASS backend handles the checkpoint's non-standard scale format correctly. The entire debugging trajectory pivots on the evidence retrieved in this single grep command.

Conclusion

Message 185 is a textbook example of why forensic log analysis matters in systems debugging. The assistant had spent multiple iterations chasing a plausible but incorrect hypothesis, changing configuration parameters and restarting servers without ever examining the actual error content. The simple act of grepping for the scheduler exception and reading the three preceding lines — the assertion message from the CUDA kernel — provided the evidence needed to reframe the problem entirely. It transformed a vague "device-side assert triggered" into a specific diagnosis: "probability tensor contains NaN/Inf." That specificity is what made the solution possible.