The Moment of Insight: Tracing a NaN Crash to an Auto-Selected KV Cache

In the high-stakes world of deploying cutting-edge large language models on novel GPU architectures, the debugging process often resembles detective work. Each failed experiment produces a clue, each log line a piece of evidence. Message 206 in this opencode session represents a pivotal moment—a flash of insight where the assistant finally understands why its carefully crafted configuration changes had no effect. The message is brief, almost anticlimactic in its simplicity, but it encapsulates the entire arc of a multi-hour debugging saga.

The Message

Here is the complete message, exactly as it appears in the conversation:

[assistant] I see — it still auto-set kv_cache_dtype='fp8_e4m3' even though I didn't pass it! The DSA model auto-selects fp8 KV cache. And it still uses nsa_decode_backend='flashmla_kv'. Let me explicitly override both: [bash] ssh 10.1.230.175 'pkill -9 -f sglang; sleep 2'

On its surface, this is a simple observation followed by a command to kill the server. But to understand why this message matters, we must trace the long chain of failures that led to it.

The Context: A Persistent NaN Crash

The assistant had been attempting to deploy the GLM-5-NVFP4 model—a massive Mixture-of-Experts (MoE) language model quantized to 4-bit floating point (NVFP4)—on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. Each GPU boasts 96 GB of VRAM, and the system has approximately 768 GB total. The model uses the glm_moe_dsa architecture, where DSA stands for DeepSeek Sparse Attention, a sophisticated attention mechanism designed for efficient long-context processing.

The deployment had been failing consistently. The server would load the model successfully, capture CUDA graphs during warmup, and then crash on the very first decode request with a devastating error: Assertion 'probability tensor contains either inf, nan or element < 0' failed. This device-side assert triggered error meant that somewhere in the model's forward pass, the logits—the raw probability scores the model uses to predict the next token—had degenerated into NaN (Not a Number) or infinity values.

The assistant had tried numerous remedies. It switched attention backends from flashinfer to triton to flashmla_sparse. It forced the FP8 GEMM backend to cutlass to avoid DeepGemm, which had emitted a warning about incompatible scale formats. It disabled CUDA graphs. It consulted the local research repository (FINDINGS.md), which documented a successful deployment of a similar model (Kimi K2-Thinking NVFP4) on the same hardware. Nothing worked. The NaN crash was relentless.

The False Dawn: Message 204–205

In message 204, the assistant launched a new server instance with what it believed were two key changes: it removed --kv-cache-dtype fp8_e4m3 (expecting the KV cache to default to BF16) and added --disable-cuda-graph. The reasoning was sound: if the KV cache used BF16 instead of FP8, the NSA (Neural Sparse Attention) backends wouldn't be forced to use flashmla_kv, a kernel path that might be producing the NaN values.

The assistant waited, monitored the logs, and in message 205 discovered the truth:

[2026-02-19 00:19:10] WARNING server_args.py:1177: Set NSA backends for fp8_e4m3 KV Cache: prefill=flashmla_auto, decode=flashmla_kv.

Despite not passing --kv-cache-dtype, the server had auto-selected fp8_e4m3 KV cache anyway. The DSA model architecture was overriding the default. The assistant's careful configuration change had been silently ignored.

The Insight (Message 206)

This is where message 206 becomes the fulcrum of the entire debugging session. The assistant connects the dots:

The KV cache dtype was not defaulting to BF16. The DSA model forces fp8_e4m3 regardless of what the user passes (or doesn't pass). This is an architectural constraint built into the model's configuration within sglang's server code.
Because the KV cache was FP8, the NSA backends were forced to flashmla_kv. The warning from message 205 made this explicit: "Set NSA backends for fp8_e4m3 KV Cache: prefill=flashmla_auto, decode=flashmla_kv."
The NaN crash might be caused by the flashmla_kv decode backend on Blackwell SM120 GPUs. The assistant had already seen evidence that flashmla_kv was problematic—it had tried switching to triton and flashmla_sparse earlier, but those attempts didn't address the root cause because the KV cache dtype was still FP8.
The solution required explicit overrides. Simply omitting the flag wasn't enough. The assistant needed to pass --kv-cache-dtype auto to explicitly request non-FP8 KV cache, and --nsa-decode-backend trtllm / --nsa-prefill-backend trtllm to explicitly choose a different NSA backend. The message ends with pkill -9 -f sglang; sleep 2—killing the failed server instance to prepare for the next attempt. This is not defeat; it is the reset before the next experiment, informed by a deeper understanding of the system.

Assumptions and Their Violations

The debugging process up to this point had been guided by several assumptions, most of which were reasonable but ultimately incorrect:

Assumption 1: Omitted flags use defaults. The assistant assumed that not passing --kv-cache-dtype would let the server use its default value (BF16). This is how most command-line tools work. But sglang's server architecture has model-specific overrides: the DSA model's configuration forces FP8 KV cache, overriding both the default and any user omission.

Assumption 2: The NaN was caused by DeepGemm. The warning about DeepGemm being enabled with a non-ue8m0 scale format was a compelling suspect. The assistant spent multiple rounds trying to disable DeepGemm via --fp8-gemm-backend cutlass. While this might have been a contributing factor, the persistence of the crash after disabling DeepGemm pointed to a different root cause.

Assumption 3: The attention backend choice was independent of KV cache dtype. The assistant had tried switching attention backends earlier, but those attempts didn't change the KV cache dtype. The realization in message 206 is that the KV cache dtype and the NSA backend selection are coupled: FP8 KV cache forces flashmla_kv decode backend.

Input Knowledge Required

To understand this message, one needs knowledge of several technical domains:

KV cache in transformer inference: The key-value cache stores attention activations from previous tokens to avoid recomputation. Its dtype affects both memory usage and numerical behavior.
FP8 vs BF16 precision: FP8 (8-bit floating point) uses half the memory of BF16 but has less numerical range and precision, making it more susceptible to overflow and NaN issues.
DSA/NSA attention: DeepSeek Sparse Attention is a family of sparse attention mechanisms that use specialized CUDA kernels (flashmla_kv, trtllm) for efficient decoding.
Blackwell SM120 architecture: NVIDIA's Blackwell GPU architecture (compute capability SM120) has specific requirements and limitations for CUDA kernels, which may not all be compatible.
sglang server architecture: The server has model-specific configuration overrides that can silently change user-provided arguments.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

DSA models force FP8 KV cache in sglang. This is an architectural constraint that future deployers must account for.
Explicit overrides are necessary. To change the KV cache dtype for DSA models, one must pass --kv-cache-dtype auto (or another explicit value) rather than relying on defaults.
NSA backends must be explicitly specified. When overriding the KV cache dtype, the NSA decode and prefill backends must also be explicitly set to avoid the auto-selected flashmla_kv path.
The debugging methodology is validated. The assistant's approach of checking logs after each change, even when the change seemed straightforward, caught the silent override that would otherwise have gone unnoticed.

The Thinking Process

The reasoning visible in this message reveals a methodical debugging mind at work. The assistant had just received the log output from message 205 showing that kv_cache_dtype was still fp8_e4m3 despite not being passed. The "I see" at the beginning of the message is the moment of cognitive synthesis—the pieces click together.

The assistant could have simply killed the server and tried something else randomly. Instead, it explicitly articulates the two things it learned: (1) the DSA model auto-selects fp8 KV cache, and (2) it still uses flashmla_kv decode backend. This explicit articulation is crucial because it transforms a vague intuition ("my change didn't work") into a specific hypothesis ("the model overrode my setting") that can be tested in the next iteration.

The decision to "explicitly override both" is the logical conclusion of this insight. The assistant will now pass --kv-cache-dtype auto and --nsa-decode-backend trtllm / --nsa-prefill-backend trtllm in the next launch (message 207), directly confronting the auto-selection mechanism rather than hoping to sidestep it.

Broader Significance

This message exemplifies a pattern that appears throughout complex systems debugging: the moment when a developer realizes that the system is not behaving as documented or assumed. The auto-selection of FP8 KV cache by the DSA model is not a bug—it is likely an intentional optimization, since FP8 KV cache saves memory and DSA models are designed to work with it. But for the assistant trying to isolate a NaN crash, this optimization becomes an obstacle that must be understood and worked around.

The message also highlights the importance of reading logs carefully. The warning in message 205—"Set NSA backends for fp8_e4m3 KV Cache"—was not an error; it was an informational message. But it contained the key to understanding why all previous attempts had failed. The assistant's diligence in checking the logs, even after what seemed like a straightforward configuration change, paid off with this crucial insight.

In the end, message 206 is a testament to the iterative nature of debugging at the frontier of AI infrastructure. Each failure teaches something new, and the most valuable insights often come not from the crashes themselves, but from the quiet discrepancies between what we expect and what the system actually does.