The Breakthrough: How trtllm NSA Backends Unlocked GLM-5-NVFP4 on Blackwell SM120 GPUs

A Pivotal Moment in a Debugging Odyssey

Message 218 (assistant): "Excellent! Everything is working perfectly: - Correct code output with docstring, type hints, proper is_prime implementation - Clean reasoning chain — the model thinks through the problem step by step - No errors in server log — clean HTTP 200 responses - Decode throughput: ~12.5 tokens/sec (single request, no CUDA graph, no tuning yet)

>

The key fix was using --nsa-decode-backend trtllm --nsa-prefill-backend trtllm — the other NSA backends (flashmla_kv, flashmla_sparse) produce NaN on SM120, but trtllm works.

>

Now let me move on to tuning and load testing."

This short message, delivered with the understated enthusiasm of "Excellent! Everything is working perfectly," marks the culmination of a grueling debugging session that spanned multiple hours and dozens of failed attempts. The assistant had been locked in a battle against a persistent NaN (Not-a-Number) crash that struck every time the GLM-5-NVFP4 model attempted to generate more than a handful of tokens during the decode phase. The solution, when it finally arrived, was deceptively simple: two command-line flags, --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm, that forced the model's sparse attention mechanism to use a specific backend implementation. But behind that simple fix lay a deep understanding of GPU architecture, attention kernel design, and the subtle incompatibilities between software and hardware.

The Context: A Model That Wouldn't Speak

To understand the significance of message 218, one must first understand the problem it solved. The GLM-5-NVFP4 is a massive 744-billion-parameter mixture-of-experts (MoE) model, with 256 experts and 40 billion active parameters per token. It uses a novel architecture called DeepSeek Sparse Attention (DSA), which forces the use of NSA (Native Sparse Attention) backends in SGLang, the inference serving framework. The model had been quantized to NVFP4 format, a 4-bit floating-point quantization scheme, and was being deployed on eight NVIDIA RTX PRO 6000 Blackwell GPUs — specifically the SM120 architecture variant, not the datacenter SM100 Blackwell chips.

The assistant had been systematically working through a list of seven different configuration attempts, each documented in the conversation's context. Attempts 1 through 6 had all ended the same way: the server would load successfully, weights would be distributed across the eight GPUs, KV cache would be allocated, the prefill warmup would pass, and then — during decode — the model would crash with a catastrophic error: "Assertion probability tensor contains either inf, nan or element < 0 failed. torch.AcceleratorError: CUDA error: device-side assert triggered." The model was producing garbage, and the root cause was a mystery with multiple suspects.

The suspects included: DeepGemm kernel incompatibility with the model's non-standard scale format, Transformers 5.2.0 RoPE parameter incompatibilities, and most likely, the NSA attention backends themselves being untested or broken on the SM120 architecture. The SM120 GPUs have only 101KB of shared memory compared to 228KB on SM100, which had already caused issues with Triton attention kernels that were fixed by a prior pull request (PR #14311). But the NSA backends presented a different class of problem — they were not crashing with out-of-memory errors but were silently producing NaN values in the probability tensors.

The Breakthrough: Attempt 7

Attempt 7, which is the subject of message 218, was launched with a specific set of parameters that the assistant had carefully selected based on the pattern of failures. The critical flags were:

--nsa-decode-backend trtllm
--nsa-prefill-backend trtllm
--fp8-gemm-backend cutlass
--disable-cuda-graph

The key insight was to use the trtllm backend for NSA operations rather than the default flashmla_kv or flashmla_sparse backends. The trtllm backend refers to NVIDIA's TensorRT-LLM library, which provides optimized implementations of attention kernels. On the SM120 architecture, where shared memory is constrained and certain CUDA capabilities differ from SM100, the TensorRT-LLM path apparently handles the sparse attention computation correctly, while the flashmla-based backends produce numerical instability.

The message reports three successful inference tests that confirm the fix. The first test, a simple "What is 2+2?" query with 64 max tokens, produced a coherent reasoning chain. The second test, with 256 max tokens, produced the correct answer "4" and completed naturally with finish_reason: "stop". The third test, a more demanding request to write a Python prime-checking function with 1024 max tokens, produced correct, well-formatted code with docstrings and type hints. The server log showed no errors — just clean HTTP 200 responses.

The Thinking Process: What the Message Reveals

The assistant's thinking in this message is revealed through several layers. First, there is the explicit analysis: the assistant lists four bullet points confirming different aspects of success — correct code output, clean reasoning chain, no server errors, and a baseline throughput measurement. This structured verification shows a methodical approach to validation: the assistant doesn't just check that the model doesn't crash, but verifies that the output is semantically correct, that the reasoning process is coherent, that the server infrastructure is healthy, and that performance is measurable.

The throughput measurement of ~12.5 tokens per second for a single request is particularly revealing. The assistant notes this as a baseline: "no CUDA graph, no tuning yet." This indicates an awareness that the current performance is not optimized — the assistant has already formed a mental model of what tuning steps will follow, including enabling CUDA graph optimization, adjusting memory fractions, and running parallel load tests. The message is not an endpoint but a milestone; the assistant is already looking ahead to the next phase.

The todo list embedded in the message confirms this forward-looking orientation. The first item ("Test Attempt 7") is marked completed, but the second and third items ("Tune sglang params for parallel query throughput" and "Run load tests with sglang.bench_serving") are marked in progress and pending respectively. This todo list structure reveals the assistant's planning process: the debugging phase had a clear exit criterion (successful inference without NaN), and once that criterion was met, the assistant immediately pivoted to the performance optimization phase.

Input Knowledge Required

Understanding this message requires substantial domain knowledge spanning several areas. First, one must understand the concept of sparse attention in large language models — the DSA architecture used by GLM-5 selectively computes attention over a subset of positions rather than all positions, which requires specialized kernel implementations. Second, one must understand the GPU architecture landscape: SM120 (RTX PRO 6000 Blackwell) versus SM100 (datacenter Blackwell) and how shared memory limits, CUDA capability versions, and kernel compatibility differ between them. Third, one must understand the SGLang inference framework's architecture, particularly how NSA backends are selected and how the --nsa-decode-backend and --nsa-prefill-backend flags control which kernel implementation is used for sparse attention operations.

The message also assumes knowledge of the debugging history. The phrase "the other NSA backends (flashmla_kv, flashmla_sparse) produce NaN on SM120" references the six failed attempts documented in the preceding conversation. Without that context, the significance of the trtllm backend choice would be lost. The reader must understand that this was not a random trial but a targeted selection based on systematic elimination of alternatives.

Output Knowledge Created

This message creates several important pieces of knowledge. First and foremost, it establishes that the GLM-5-NVFP4 model can be successfully deployed on SM120 Blackwell GPUs using SGLang with the trtllm NSA backends. This is a concrete, actionable finding for anyone attempting the same deployment. Second, it documents that the flashmla_kv and flashmla_sparse NSA backends are incompatible with SM120 — they produce NaN errors during decode. This is a negative result that saves future practitioners from repeating the same debugging effort. Third, it establishes a baseline throughput of ~12.5 tokens/second for single-request decode on this hardware configuration, which provides a reference point for optimization efforts.

The message also implicitly creates knowledge about the debugging methodology itself. The systematic approach of trying multiple configurations, documenting each attempt, and isolating the variable (NSA backend selection) that made the difference is a model for how to approach similar hardware-software compatibility issues. The assistant's decision to test with three progressively more demanding prompts (simple math, brief answer, code generation) demonstrates a validation strategy that goes beyond mere crash testing to verify semantic correctness.

Assumptions and Potential Pitfalls

The message makes several assumptions that are worth examining. It assumes that the trtllm backend will continue to work reliably under load — the tests performed were single-request, low-concurrency tests. The baseline throughput of 12.5 tokens/second is promising, but it remains to be seen whether the backend maintains numerical stability under concurrent requests with higher memory pressure and more complex scheduling.

The message also assumes that the fix is complete — that the NaN issue is fully resolved by the backend selection. However, the DeepGemm warning about scale format incompatibility that appeared in earlier attempts (message 208) was not explicitly addressed. The assistant had set --fp8-gemm-backend cutlass to avoid DeepGemm for FP8 operations, but the warning suggested DeepGemm might still be enabled for some operations. The successful tests suggest this is not causing problems in practice, but the possibility of edge cases remains.

Another assumption is that the Transformers 5.2.0 RoPE compatibility warning is not an issue. The assistant had noted earlier that downgrading to Transformers 4.57.x was not possible because it lacks the glm_moe_dsa model type. The successful inference tests validate this assumption empirically, but the warning about "issues related to RoPE parameters" could manifest in subtle ways that a simple math question and code generation test might not catch — for instance, in long-context scenarios or specific attention patterns.

The Broader Significance

Message 218 represents more than just a successful deployment — it illustrates a fundamental pattern in AI infrastructure engineering. When deploying cutting-edge models on non-standard hardware, the path from "model loads" to "model produces correct output" is often a treacherous one filled with numerical instability, kernel incompatibilities, and silent correctness failures. The NaN crash that plagued the first six attempts is a particularly insidious class of bug: the model appears to load correctly, the server appears to start successfully, and the crash only manifests during actual inference. This makes debugging difficult because the error surface is deep — the problem only appears after minutes of loading and initialization.

The solution — selecting a different backend implementation — highlights the importance of software diversity in the AI ecosystem. SGLang's support for multiple NSA backends (flashmla_kv, flashmla_sparse, trtllm) meant that when one path failed, alternatives were available. This design choice, which might seem like unnecessary complexity in a simpler world, proved essential for hardware compatibility. The trtllm backend, backed by NVIDIA's TensorRT-LLM library, apparently handles the SM120 architecture's constraints better than the flashmla-based implementations, which may have been optimized primarily for SM100 or other architectures.

The message also demonstrates the value of systematic experimentation in debugging. The assistant maintained a clear table of attempted configurations (visible in the preceding context at message 211), tracking which parameters were varied and what the result was. This structured approach allowed the assistant to isolate the NSA backend as the critical variable, rather than getting lost in the space of possible causes (DeepGemm, Transformers version, KV cache dtype, CUDA graph settings). The todo list mechanism further shows how the assistant managed the transition between debugging and optimization phases, ensuring that no steps were dropped.

Conclusion

Message 218 captures the moment when a complex, multi-hour debugging effort finally paid off. The solution — using --nsa-decode-backend trtllm --nsa-prefill-backend trtllm — was specific and technical, but the thinking behind it was anything but narrow. It required understanding GPU architecture differences, attention kernel implementations, the SGLang framework's backend selection mechanism, and the systematic elimination of alternative hypotheses. The message is simultaneously a victory lap and a launch pad: the assistant celebrates the successful fix while already planning the next phase of performance optimization. For anyone following this conversation, it provides both a concrete solution to a specific deployment problem and a model for how to approach similar debugging challenges in the rapidly evolving landscape of AI hardware and software.