The Breakthrough: How trtllm NSA Backends Rescued GLM-5-NVFP4 on SM120 Blackwell GPUs

A Moment of Relief After Hours of Debugging

In any complex engineering effort, there comes a moment when everything clicks into place — when a persistent, frustrating blocker finally yields to the right combination of parameters. Message 216 of this opencode session captures exactly such a moment. After seven failed attempts spanning multiple hours of debugging, the assistant types three words that carry the weight of a significant breakthrough: "IT WORKS!"

The message reads:

IT WORKS! The trtllm NSA backend is producing coherent output! The model is reasoning correctly about 2+2. It hit max_tokens: 64 before finishing its reasoning chain, but the output is valid — no NaN, no garbage, no crash.

>

Let me send a follow-up request with more tokens to confirm it can complete a full response: [bash] ssh 10.1.230.175 'curl -s --max-time 120 http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"glm-5\", \"messages\": [{\"role\": \"user\", \"content\": \"What is 2+2? Answer briefly.\"}], \"max_tokens\": 256, \"temperature\": 0.0}" 2>&1' {"id":"bcef00a3d5924e0c8399d35d8cd24b8d","object":"chat.completion","created":1771460641,"model":"glm-5","choices":[{"index":0,"message":{"role":"assistant","content":"4","reasoning_content":"The user is asking a very simple math question: \"What is 2+2?\".\nThe constraint is to \"Answer briefly\".\n\n1. Analyze the request:\n Question: 2 + 2\n Constraint: Brief answer.\n\n2. Calculate the answer:\n 2 + 2 = 4.\n\n3. Formulate the response:\n Option A: \...

This message is deceptively simple — a short exclamation followed by a curl command and its JSON response. But beneath this surface lies a rich story of systematic debugging, hardware-specific knowledge, and the kind of inference engineering that makes large language model deployment on novel hardware possible.

The Context: A Seven-Attempt Debugging Marathon

To understand why this message matters, one must appreciate the wall the assistant had been beating against. The GLM-5-NVFP4 model is a 744-billion-parameter Mixture-of-Experts model with 256 experts (8 activated per token), quantized to NVFP4 (NVIDIA's 4-bit floating point format). It uses DeepSeek Sparse Attention (DSA), a novel attention mechanism that forces the use of "NSA" (likely "Non-Scalable Attention" or a similar internal abbreviation) attention backends in SGLang, the inference serving framework.

The hardware is equally unusual: 8x NVIDIA RTX PRO 6000 Blackwell GPUs — but these are SM120 architecture, not the datacenter SM100 Blackwell that the model card was designed for. SM120 has only 101KB of shared memory versus 228KB on SM100, a difference that breaks many CUDA kernels designed for the datacenter variant.

Every previous attempt (documented as Attempts 1 through 6 in the assistant's notes) had failed with the same catastrophic error during decode:

Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
torch.AcceleratorError: CUDA error: device-side assert triggered

This is a NaN/Inf in logits — the model produces garbage after generating a few tokens. The assistant had tried:

Why trtllm Was the Right Choice

Attempt 7, which produced the successful result in message 216, used --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm. The "trtllm" backend refers to NVIDIA's TensorRT-LLM integration in SGLang — a path that uses NVIDIA's optimized inference library rather than the custom CUDA kernels that SGLang normally uses for NSA attention.

The key insight is that the other NSA backends (flashmla_kv, flashmla_sparse) are built on custom CUDA kernels that were likely optimized for SM100 datacenter Blackwell GPUs. On SM120, these kernels may have subtle bugs — incorrect shared memory sizes, wrong warp configurations, or incompatible instruction sequences — that produce numerical instability (NaN). The TensorRT-LLM backend, being a more mature and thoroughly tested library from NVIDIA, likely handles the SM120 architecture correctly because it has been validated across a wider range of GPU architectures.

This is a common pattern in ML inference engineering: when custom CUDA kernels fail on novel hardware, falling back to the vendor's optimized library often provides a working path, even if it may not be the fastest option.

Assumptions Made and Corrected

Several assumptions had to be challenged to reach this solution:

Assumption 1: The NaN was caused by DeepGemm or FP8 quantization. Earlier attempts focused on disabling DeepGemm (via --fp8-gemm-backend cutlass) and changing KV cache dtype. While these were reasonable suspects, the real culprit was the NSA attention backend itself.

Assumption 2: SM120 and SM100 are interchangeable for NSA kernels. The model card's configuration was designed for SM100. The assistant initially assumed the same NSA backends would work on SM120, but the shared memory difference (101KB vs 228KB) made many custom kernels unstable.

Assumption 3: The trtllm backend might not be available or might be SM100-only. The assistant had noted earlier that "the flashinfer_trtllm path was found to be SM100-only" (from the chunk summary). But the trtllm NSA backend (distinct from flashinfer_trtllm) proved to be available and functional on SM120.

Assumption 4: The RoPE incompatibility with Transformers 5.2.0 was the root cause. The server log had warned about Transformers 5.2.0 RoPE parameter issues, and the assistant had considered this a prime suspect. But the trtllm backend worked despite this warning, showing the RoPE issue was not the NaN source.

Input Knowledge Required

To understand and appreciate this message, one needs knowledge spanning several domains:

GPU Architecture: Understanding that SM120 (RTX PRO 6000 Blackwell) and SM100 (datacenter Blackwell) are different architectures with different shared memory capacities, and that CUDA kernels must be compiled for the specific SM version.

Attention Mechanisms: Knowledge of DeepSeek Sparse Attention (DSA) and how it forces NSA backends in SGLang, and the distinction between NSA prefill and decode backends.

SGLang Inference Stack: Familiarity with SGLang's server arguments (--nsa-decode-backend, --nsa-prefill-backend, --fp8-gemm-backend, --attention-backend) and how they interact with model architecture.

Quantization Formats: Understanding NVFP4 (NVIDIA's 4-bit floating point) and how it differs from standard FP8 or INT4 quantization, and that the model forces fp8_e4m3 KV cache regardless of user settings.

The Debugging History: The seven previous attempts, each documented with specific parameter combinations and their failure modes, provide the experimental record that makes this success meaningful.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. A working configuration for GLM-5-NVFP4 on SM120 GPUs: --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --fp8-gemm-backend cutlass --disable-cuda-graph combined with the standard model card settings.
  2. Validation that TensorRT-LLM NSA backends work on SM120: This is non-obvious — the flashinfer_trtllm path was known to be SM100-only, but the standalone trtllm NSA backend works on SM120.
  3. Confirmation that the NaN decode crash was an NSA backend issue, not a quantization or RoPE issue: This narrows the debugging space for future deployments of DSA models on SM120 hardware.
  4. A baseline for further tuning: With the model now producing coherent output, the assistant can proceed to throughput optimization and load testing.

The Thinking Process

The message reveals a careful, methodical approach. The assistant doesn't just celebrate — it immediately designs a follow-up test with more tokens (max_tokens: 256) to confirm the model can complete a full response rather than just producing a partial reasoning chain. This shows an understanding that the first test (which hit max_tokens: 64) might have succeeded only for the prefill/reasoning phase, and the real test is whether the model can generate a complete answer including the final content token.

The choice of prompt ("What is 2+2? Answer briefly.") is also deliberate — it's a trivial question that any functional model should answer correctly. A simple "4" response with a reasoning chain is exactly what one would expect from a working GLM-5 model. This is a minimal sanity check, not a comprehensive evaluation.

The JSON response confirms the model produced "4" as the content, with a reasoning chain showing it analyzed the request, calculated the answer, and formulated a response. The fact that the reasoning was truncated (ending with "Option A: \...") is expected given the max_tokens: 256 limit and the verbose reasoning style of GLM models.

Implications and Next Steps

With the NaN decode crash resolved, the assistant can now proceed to the next phases of the deployment: tuning server parameters for parallel query throughput, enabling CUDA graphs (which were disabled as a precaution), and running load tests with sglang.bench_serving. The chunk summary confirms this trajectory — the assistant went on to baseline throughput benchmarks, tune memory fractions, and diagnose PCIe P2P latency bottlenecks.

This message represents the inflection point of the entire deployment effort. Before it, the team was stuck in a debugging loop, unable to get any useful output from the model. After it, the path forward to a production-ready deployment becomes clear. In the narrative of this coding session, message 216 is the turning point — the moment when persistence, systematic experimentation, and deep hardware knowledge finally paid off.