The Breakthrough: How trtllm NSA Backends Rescued GLM-5-NVFP4 on SM120 Blackwell GPUs

In any complex engineering effort, there comes a single moment when a persistent, frustrating problem finally breaks. For this opencode coding session—a multi-hour odyssey to deploy the massive GLM-5-NVFP4 (744B parameter MoE model) across 8 NVIDIA RTX PRO 6000 Blackwell GPUs—that moment arrived in message [msg 215]. After seven failed attempts, each ending in the same catastrophic NaN crash during decode, the assistant finally obtained coherent model output. This article examines that single message in depth: why it was written, the reasoning that led to it, the knowledge it required, the knowledge it produced, and the assumptions—both correct and incorrect—that shaped it.

The Message Itself

The message is deceptively simple—a single bash command piped through SSH to the remote inference server, followed by its JSON response:

``bash ssh 10.1.230.175 'curl -s --max-time 120 http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"glm-5\", \"messages\": [{\"role\": \"user\", \"content\": \"What is 2+2? Answer briefly.\"}], \"max_tokens\": 64, \"temperature\": 0.0}" 2>&1' ``

>

``json {"id":"840bda2ae8ef401c88c2988278cd4e98","object":"chat.completion","created":1771460620,"model":"glm-5","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":"The user is asking a very simple math question: \"What is 2+2?\".\nThe constraint is to \"Answer briefly\".\n\n1. **Analyze the request:**\n * Question: 2 + 2\n * Constraint: Brief answer.\n\n2. **Calculate the answer:","tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":... ``

At first glance, this looks like any other successful API call to a running LLM server. But to understand why this message represents a genuine breakthrough, one must grasp the wall of failures that preceded it.

Why This Message Was Written: The NaN Wall

The message was written to answer a single, urgent question: Does Attempt 7 work? The context leading up to this moment is a story of iterative debugging against a particularly vicious class of bug—a silent numerical corruption that only manifests during the decode phase of transformer inference.

The assistant had been trying to deploy GLM-5-NVFP4, a 744B-parameter Mixture-of-Experts model quantized to NVFP4 precision, using sglang's inference server. The model uses DeepSeek Sparse Attention (DSA), which forces the use of specialized "NSA" (Native Sparse Attention) backends for the attention computation. Every prior attempt—six in total—had ended the same way: the server would load successfully, distribute ~62GB of weights per GPU, allocate KV cache, pass the prefill warmup phase, and then crash during decode with the error:

Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
torch.AcceleratorError: CUDA error: device-side assert triggered

This is a catastrophic failure. The model's logits—the probability distribution over the vocabulary—contain NaN or Inf values, meaning the model has gone numerically unstable. The assistant had systematically tried different combinations of attention backends (flashinfer, triton), NSA decode backends (flashmla_kv, flashmla_sparse), FP8 GEMM backends (auto, cutlass), and CUDA graph settings, as documented in the configuration table at [msg 211]. Every combination crashed.

The critical insight that led to Attempt 7 was the distinction between SM100 (datacenter Blackwell, with 228KB shared memory) and SM120 (RTX PRO 6000 Blackwell, with only 101KB shared memory). The model card on HuggingFace recommended a configuration designed for SM100, and the flashmla_kv and flashmla_sparse NSA backends appeared to rely on shared memory or other SM100-specific features that were absent or behaved differently on SM120. The trtllm backend—which uses TensorRT-LLM's attention kernels—might use a completely different code path that doesn't trigger the SM120-specific issue.

This message was therefore the moment of truth for that hypothesis.

How the Decision Was Made: The Path to trtllm

The decision to try trtllm NSA backends was not arbitrary. It emerged from a process of elimination. The assistant had already ruled out:

1. DeepGemm as the sole culprit: The warning "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0" appeared in all attempts, but switching to --fp8-gemm-backend cutlass (as in Attempt 5 and 6) didn't fix the NaN, suggesting DeepGemm wasn't the primary cause.
2. CUDA graphs as the trigger: Attempt 6 used --disable-cuda-graph but still crashed, ruling out CUDA graph capture as the source of numerical instability.
3. KV cache dtype as the root cause: The model auto-selects fp8_e4m3 KV cache regardless of the --kv-cache-dtype flag (as discovered in [msg 206]), but the NaN persisted across attempts with and without explicit dtype control.
4. Transformers version as the primary issue: While the warning about transformers 5.2.0 RoPE incompatibility was concerning, the NaN pattern (clean prefill, corrupted decode) pointed more toward an attention backend issue than a parameter loading issue. By elimination, the NSA attention backends became the prime suspect. The question was: which NSA backend works on SM120? The assistant had tried flashmla_kv (the default) and flashmla_sparse. The remaining option was trtllm. The server was launched with --nsa-decode-backend trtllm --nsa-prefill-backend trtllm in [msg 207], and after waiting for the model to load and warm up (monitored in [msg 208] and [msg 209]), the assistant finally tested it in this message.

Input Knowledge Required

To understand this message, one needs substantial context spanning multiple domains:

• GPU architecture: The difference between SM100 (datacenter Blackwell, 228KB shared memory) and SM120 (RTX PRO 6000 Blackwell, 101KB shared memory) is critical. The flashmla attention kernels were likely optimized for SM100's larger shared memory and may silently produce incorrect results when shared memory is insufficient.
• SGLang's attention backend architecture: SGLang supports multiple attention backends (flashinfer, triton, flashmla, trtllm). The DSA model forces NSA backends specifically, and each NSA backend (flashmla_kv, flashmla_sparse, trtllm) implements the sparse attention pattern differently.
• NVFP4 quantization: The model uses NVIDIA's NVFP4 format for MoE expert weights. This quantization format interacts with the attention computation in complex ways, and the scale format warning (ue8m0 vs non-ue8m0) suggests potential accuracy issues.
• The NaN crash pattern: The specific error—a device-side assert on probability tensors containing NaN/Inf—indicates that the numerical instability occurs in the final softmax or sampling step, but the root cause could be anywhere upstream in the attention computation.
• The prior research repository: The assistant had access to /home/theuser/glm-kimi-sm120-rtx6000bw/FINDINGS.md, an 871-line document documenting prior deployments of GLM-4.7 and Kimi K2 models on the same SM120 hardware. This document contained the critical observation that Kimi K2-Thinking NVFP4 (which does NOT use DSA) worked successfully, while GLM-5 (which DOES use DSA) was the first model to trigger the NSA backend issue.

Output Knowledge Created

This message produced several pieces of valuable knowledge:

1. trtllm NSA backends work on SM120: This is the primary finding. The model produced coherent reasoning content—"The user is asking a very simple math question: 'What is 2+2?'" —indicating that the attention computation is numerically stable with the trtllm backend.
2. The NaN crash is definitively an NSA backend issue: By process of elimination, the flashmla_kv and flashmla_sparse backends are incompatible with SM120 GPUs for this model, while trtllm is compatible.
3. The model is deployable on RTX PRO 6000 Blackwell: This unblocks the entire deployment pipeline. The assistant can now proceed to tuning, benchmarking, and load testing.
4. A working configuration baseline: The successful configuration (--nsa-decode-backend trtllm --nsa-prefill-backend trtllm combined with --fp8-gemm-backend cutlass --disable-cuda-graph) becomes the foundation for all further optimization. However, the message also reveals that the model's output is truncated at 64 tokens (the max_tokens parameter) and only shows the beginning of reasoning. The response has "finish_reason":"length", meaning the model was cut off mid-reasoning. This is expected for a quick test, but it means the message doesn't confirm that the model produces correct final answers—only that it doesn't crash and produces sensible-looking reasoning.

Assumptions and Potential Pitfalls

Several assumptions underpin this message:

• The test prompt is sufficient: The assistant assumes that a simple "What is 2+2?" query is representative. It's possible that the trtllm backend works for short, simple prompts but fails on longer or more complex inputs. The 64-token limit is also very short—longer generations might still trigger numerical instability.
• The trtllm backend is available: The message assumes that the trtllm NSA backend was compiled and installed correctly in the sglang build. If it wasn't, the server would have failed to start or would have silently fallen back to a different backend.
• No silent correctness issues: The model produces reasoning text, but the message doesn't verify that the mathematical answer is correct. The reasoning content shown ("Calculate the answer:") is cut off, so we don't see the final answer. It's possible that the model produces plausible-sounding but incorrect output—a different failure mode than NaN crashes, but still a failure.
• The fix is generalizable: The assumption is that trtllm works for all DSA models on SM120, not just this specific checkpoint. This would need to be verified with other models. One notable mistake in the surrounding context is the initial assumption that the DeepGemm scale format warning was the primary issue. The assistant spent several attempts switching FP8 GEMM backends (Attempts 5 and 6) before focusing on NSA backends. While not wasted effort—it helped rule out DeepGemm—it did delay the eventual solution.

The Thinking Process Revealed

The message reveals a structured, hypothesis-driven debugging approach. The assistant:

1. Formulates a clear test: A single curl command to the running server with a simple prompt, short max_tokens, and zero temperature (for deterministic output).
2. Sets appropriate timeouts: The --max-time 120 flag gives the server 120 seconds to respond, accounting for the fact that first-token latency on a 744B model across 8 PCIe-connected GPUs can be significant.
3. Checks the response carefully: Rather than just checking for a 200 HTTP status code, the assistant reads the actual response content, noting the reasoning_content field.
4. Interprets the result correctly: The presence of coherent reasoning content is immediately recognized as success, contrasting with the previous NaN crashes. The thinking process visible in the broader context (messages [msg 204] through [msg 214]) shows a systematic narrowing of hypotheses. The assistant maintains a configuration table, tracks which variables changed between attempts, and uses each failure to eliminate possible causes. This is textbook scientific debugging applied to ML infrastructure.

Broader Significance

This message represents more than just a single deployment milestone. It demonstrates that:

• GPU architecture compatibility is a first-class concern: Models and frameworks designed for datacenter GPUs (SM100) may fail silently on workstation/professional GPUs (SM120) due to differences in shared memory, register count, or other microarchitectural features.
• Attention backends are not interchangeable: Different implementations of the same mathematical operation (sparse attention) can have dramatically different numerical behavior on the same hardware. The flashmla_kv and trtllm backends both implement DSA attention, but only one works on SM120.
• The value of systematic elimination: When faced with a complex failure mode involving multiple interacting components (quantization, attention, GEMM operations, CUDA graphs), the methodical approach of changing one variable at a time and maintaining a configuration table is essential. The message also opens the door to the next phase of the project: performance tuning. With the model now producing valid output, the assistant can proceed to optimize memory fractions, enable CUDA graphs (if they now work), tune MoE runner backends, and benchmark throughput—all of which are documented in the subsequent chunk of the conversation. In conclusion, message [msg 215] is a quiet triumph—a single curl command that, after hours of debugging and seven failed attempts, finally returns a JSON response without a crash. It is a testament to the value of structured debugging, domain knowledge about GPU architectures, and the patience to systematically test every combination until the right one is found.