The Art of the Iterative Launch: Debugging NaN Crashes on Blackwell GPUs

In the world of deploying large language models on cutting-edge hardware, a single command can encapsulate hours of debugging, multiple failed hypotheses, and a deep understanding of the intricate dance between model architecture, quantization schemes, and GPU kernel backends. Message 207 in this opencode session is precisely such a moment — a carefully crafted SGLang server launch command that represents the culmination of an iterative debugging journey against one of the most elusive problems in ML engineering: a NaN crash during decode on NVIDIA's Blackwell (SM120) architecture.

The Message in Full

The assistant issues a single bash command via SSH to a remote machine running Ubuntu 24.04 with 8 RTX PRO 6000 Blackwell GPUs:

ssh 10.1.230.175 'nohup bash -c '\''
export PATH=$HOME/ml-env/bin:$PATH
export CUDA_HOME=/usr/local/cuda-12.8
export NCCL_IB_DISABLE=1
export NCCL_P2P_LEVEL=PHB
export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1
export NCCL_MIN_NCHANNELS=8
export OMP_NUM_THREADS=8
export SAFETENSORS_FAST_GPU=1

python3 -m sglang.launch_server \
  --model lukealonso/GLM-5-NVFP4 \
  --served-model-name glm-5 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --trust-remote-code \
  --tp 8 \
  --mem-fraction-static 0.85 \
  --max-running-requests 64 \
  --kv-cache-dtype auto \
  --quantization modelopt_fp4 \
  --attention-backend flashinfer \
  --fp8-gemm-backend cutlass \
  --nsa-decode-backend trtllm \
  --nsa-prefill-backend trtllm \
  --moe-runner-backend flashinfer_cutlass \
  --disable-custom-all-reduce \
  --enable-flashinfer-allreduce-fusion \
  --disable-cuda-graph \
  --host 0.0.0.0 \
  --port 8000
'\'' > ~/sglang-glm5.log 2>&1 &
echo "PID: $!"'
PID: 22711

On the surface, this is a routine server launch. But every parameter choice tells a story of what went wrong before and what the assistant hypothesizes will fix it.

The Context: A Cascade of Crashes

To understand why this message was written, we must trace the debugging trail that led here. The session had been attempting to deploy the GLM-5-NVFP4 model — a quantized (FP4) Mixture-of-Experts model from the GLM family — on 8 Blackwell GPUs using SGLang. The model uses a novel architecture called glm_moe_dsa (DeepSeek Sparse Attention), which introduces additional complexity beyond standard transformer architectures.

The previous attempts (messages 180–206) had all failed with the same catastrophic error:

Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.

This device-side assert triggered crash occurred during the decode phase, after the model loaded successfully and CUDA graph capture completed. The server would start, accept requests, and then immediately crash when trying to generate the first token. The NaN/Inf in the probability tensor meant that the model's output logits were numerically corrupted — a particularly insidious bug because it doesn't manifest during model loading or prefill, only during the autoregressive decode loop.

The Reasoning Process: What Was Tried Before

The assistant's debugging journey reveals a systematic process of elimination. Each hypothesis about the root cause was tested by changing specific configuration parameters:

Hypothesis 1: SM120 shared memory bug. The initial install used SGLang 0.5.8.post1, which lacked the SM120 block size fix (PR #14311). The assistant rebuilt from the main branch to include this fix. When the crash persisted, this hypothesis was ruled out.

Hypothesis 2: Attention backend incompatibility. The assistant tried flashinfer, triton, and flashmla_sparse attention backends. All produced the same NaN crash. This ruled out a simple attention kernel issue.

Hypothesis 3: DeepGemm scale format incompatibility. A warning in the logs stated: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." DeepGemm is an optimized GEMM kernel that auto-selects on Blackwell, but the GLM-5 checkpoint uses a different scale format (ue8m0 vs the expected format). The assistant tried --fp8-gemm-backend cutlass to force a different GEMM backend. The crash persisted, indicating the issue was deeper than the GEMM path.

Hypothesis 4: RoPE parameter incompatibility with Transformers 5.2.0. The model required Transformers 5.2.0 for its glm_moe_dsa architecture class, but a warning suggested RoPE parameter incompatibilities. The assistant briefly tried downgrading to Transformers 4.57.6, but the model's config couldn't be loaded without the newer version. This was a dead end.

Hypothesis 5: FP8 KV cache forcing problematic NSA backends. The DSA model architecture auto-selects FP8 KV cache (fp8_e4m3), which in turn forces NSA decode backends to flashmla_kv. The assistant hypothesized that the flashmla_kv backend was the source of NaN. In the previous attempt (message 204), the assistant removed --kv-cache-dtype fp8_e4m3 hoping the default would be BF16, but the server auto-selected FP8 anyway due to the DSA architecture.

What Changed in Message 207

This message represents the sixth major attempt, incorporating lessons from all previous failures. The critical changes from the previous attempt are:

1. --kv-cache-dtype auto instead of omitting the flag. Previously, the assistant had removed the explicit --kv-cache-dtype fp8_e4m3 flag, but the server auto-selected FP8 anyway. By explicitly passing auto, the assistant acknowledges that the DSA architecture will likely still select FP8, but this gives SGLang's internal logic full control over the decision rather than forcing it.

2. Explicit --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm. This is the most significant change. The assistant is explicitly overriding the NSA (Native Sparse Attention) backends to use trtllm (TensorRT-LLM) instead of the default flashmla_kv. The reasoning, as articulated in message 206, is that the NaN might be coming from the DSA attention path rather than the MoE/GEMM path. By switching to trtllm for both prefill and decode, the assistant is testing whether a different attention kernel implementation avoids the numerical instability.

3. --mem-fraction-static 0.85 reduced from 0.88. This is a subtle but important change. The memory fraction determines how much GPU memory is reserved for the model versus left for runtime allocations. Reducing it from 0.88 to 0.85 frees up more headroom, which could help if the NaN was caused by memory pressure or allocation failures during the decode phase.

4. --disable-cuda-graph retained. The previous successful deployment of Kimi K2-Thinking NVFP4 (documented in FINDINGS.md) used this flag. CUDA graph capture optimizes execution by pre-recording CUDA kernel launches, but on new architectures like Blackwell, it can introduce subtle bugs. Disabling it is a conservative choice that prioritizes correctness over performance.

Assumptions Embedded in This Message

Every parameter choice carries assumptions about the root cause of the NaN:

• The assumption that the NaN originates in the attention path, not the GEMM path. By keeping --fp8-gemm-backend cutlass (which worked in previous attempts but didn't fix the crash) and changing only the NSA backends, the assistant is betting that the attention kernels are the culprit. This is a reasonable narrowing of the hypothesis space, but it assumes the GEMM path was already ruled out.
• The assumption that trtllm backends are more numerically stable than flashmla_kv on Blackwell. TensorRT-LLM is NVIDIA's own inference optimization framework, and it may have better-tuned kernels for the SM120 architecture. However, this is an untested assumption — the assistant hasn't verified that trtllm backends work correctly with FP4 quantization and the DSA architecture.
• The assumption that the model checkpoint itself is not corrupted. The GLM-5-NVFP4 model is a quantized FP4 checkpoint. If the quantization process introduced numerical errors, no amount of backend switching would fix it. The assistant implicitly trusts the HuggingFace model repository.
• The assumption that environment variables are correctly configured. The NCCL settings (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=PHB, etc.) are carried over from previous successful runs. These were tuned for the specific GPU topology (PIX pairs across NUMA nodes) and assume the same topology holds.

Potential Mistakes and Incorrect Assumptions

While the reasoning is sound, several potential issues remain:

The trtllm backend may not support FP4 quantization. TensorRT-LLM backends for NSA attention may not be compatible with the modelopt_fp4 quantization scheme. If trtllm expects FP8 or FP16 attention computations, it could produce incorrect results or crash with a different error. The assistant is essentially trying a backend that may not have been tested with this specific model configuration.

The --kv-cache-dtype auto may still select FP8. The DSA architecture's auto-selection logic might override auto to fp8_e4m3 regardless. If the NaN is caused by FP8 KV cache precision loss interacting with the attention kernels, changing the NSA backend won't help — the KV cache dtype itself would need to be changed.

The reduction in --mem-fraction-static is a shot in the dark. There's no evidence that memory pressure caused the NaN. The error message pointed to a numerical assertion failure, not an out-of-memory error. This change may be harmless but unlikely to fix the core issue.

The assistant may be over-indexing on the DSA attention path. The NaN could still be in the GEMM path despite the DeepGemm warning being "fixed" by --fp8-gemm-backend cutlass. The cutlass backend may have its own numerical issues with FP4 quantization on Blackwell, especially for the MoE expert routing that involves complex gating logic.

Input Knowledge Required

To fully understand this message, one needs:

• Knowledge of the GLM-5-NVFP4 model architecture: It uses glm_moe_dsa (DeepSeek Sparse Attention), which is a variant of sparse attention that requires specialized NSA backends. It's quantized to FP4 using NVIDIA's ModelOpt framework.
• Knowledge of SGLang server arguments: The meaning of flags like --nsa-decode-backend, --fp8-gemm-backend, --kv-cache-dtype, and how they interact. Understanding that auto selection logic can override explicit omissions.
• Knowledge of Blackwell (SM120) architecture: The GPU's new features (FP4 support, DeepGemm, shared memory constraints) and known issues (the SM120 block size fix, DeepGemm scale format requirements).
• Knowledge of NCCL configuration: The environment variables for InfiniBand disable, P2P level, and NUMA-aware communication are specific to multi-GPU setups with specific topologies.
• Knowledge of the previous debugging context: The FINDINGS.md document that documented successful Kimi K2-Thinking NVFP4 deployment, the DeepGemm warning, the Transformers 5.2.0 RoPE warning, and the failed attempts with different attention backends.

Output Knowledge Created

This message creates several important outputs:

• A testable hypothesis: If the server starts successfully and generates coherent text, it confirms that the NaN was caused by the flashmla_kv NSA backend and that trtllm is a viable alternative on Blackwell. If it crashes again, the hypothesis is refuted and the assistant must look elsewhere.
• A documented configuration: The exact command with all parameters serves as a reproducible configuration for deploying GLM-5-NVFP4. Whether it succeeds or fails, this configuration becomes part of the knowledge base for future attempts.
• A narrowing of the search space: By explicitly setting NSA backends, the assistant is testing the boundary between SGLang's auto-selection logic and manual override. If this works, it establishes that manual backend selection is necessary for this model on this hardware.

The Thinking Process Visible in the Reasoning

The assistant's thinking, visible in the preceding messages, follows a clear scientific method:

1. Observe the symptom: NaN in probability tensor during decode.
2. Collect evidence: Log warnings about DeepGemm scale format, RoPE incompatibility, and NSA backend selection.
3. Formulate hypotheses: Each hypothesis targets a different component (attention backend, GEMM backend, KV cache dtype, Transformers version).
4. Test each hypothesis: Change one or two parameters at a time, relaunch, observe the result.
5. Refine based on results: When a change doesn't fix the crash, discard that hypothesis and move to the next. The reasoning in message 206 is particularly revealing: "I see — it still auto-set kv_cache_dtype='fp8_e4m3' even though I didn't pass it! The DSA model auto-selects fp8 KV cache. And it still uses nsa_decode_backend='flashmla_kv'. Let me explicitly override both." This shows the assistant discovering that SGLang's auto-selection logic was overriding the omission of flags, leading to the insight that explicit overrides are necessary. The PID output (PID: 22711) is the final piece — it confirms the process has started. The assistant now waits, monitors the logs, and prepares to test whether this sixth attempt finally breaks the cycle of NaN crashes. Whether it succeeds or fails, the systematic approach embodied in this single message represents the essence of debugging at the frontier of ML hardware and software.