The Moment of Truth: Testing trtllm NSA Backends for GLM-5-NVFP4 on Blackwell SM120

In the high-stakes world of deploying cutting-edge large language models on non-standard hardware, few moments are as tense as the first inference request after a long debugging session. Message [msg 214] captures exactly such a pivot point in an opencode coding session aimed at deploying the GLM-5-NVFP4 model — a 744-billion-parameter Mixture-of-Experts model with DeepSeek Sparse Attention (DSA) — across eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture) using the SGLang inference engine. The message is brief, almost casual in tone, but it represents the culmination of an extensive debugging effort spanning multiple server restarts, configuration permutations, and careful analysis of NaN crashes that had plagued every prior attempt.

The Message Itself

The assistant writes:

The server is running and says "fired up and ready to roll!" — but we haven't tested actual inference yet (just the warmup prefill). Let me test it right now to see if Attempt 7 (trtllm NSA backends) avoids the decode NaN crash.

Accompanying this is a structured todo list showing three high-priority tasks: testing Attempt 7, tuning for throughput if successful, and running load tests. The todo list reveals the contingency planning baked into this moment — the assistant has already mapped out the branching paths depending on the outcome.

Why This Message Was Written

To understand why this message exists, one must understand the debugging hell that preceded it. The GLM-5-NVFP4 model uses DeepSeek Sparse Attention (DSA), which forces SGLang to use NSA (Native Sparse Attention) backends for the attention computation. On the SM120 architecture — the RTX PRO 6000 Blackwell Server Edition — every NSA backend tested so far had produced NaN values during decode, crashing the server with a CUDA device-side assert. Attempts 1 through 6 had all failed, each eliminating one variable: flashmla_kv, flashmla_sparse, DeepGemm, CUDA graphs, fp8 KV cache. The NaN persisted.

The assistant had systematically worked through the configuration space documented in [msg 211], a comprehensive summary that reads like a battle report. The breakthrough hypothesis was that the trtllm NSA backend — which uses TensorRT-LLM kernels for attention — might handle the SM120 architecture differently than the flashmla_* backends. Attempt 7 was launched with --nsa-decode-backend trtllm --nsa-prefill-backend trtllm, and the server had started successfully, completed weight loading, passed the warmup prefill, and announced itself ready. But warmup prefill is not real inference. The true test — generating tokens beyond the first few — remained unperformed.

This message is the moment of gathering courage to run that test. The assistant is explicitly acknowledging the gap between "server started" and "inference works," and is about to cross that gap.

The Reasoning Behind Attempt 7

The choice of trtllm as the NSA backend was not arbitrary. Looking at the progression of failed attempts documented in [msg 211], the assistant had tried flashmla_kv (the auto-selected default), flashmla_sparse, and combinations with different fp8 gemm backends and CUDA graph settings. All crashed with the same NaN assertion. The trtllm backend represented a fundamentally different code path for the attention computation.

The reasoning likely went as follows: if the NaN is caused by a numerical issue specific to the flashmla kernels on SM120 — perhaps related to the reduced shared memory (101KB vs 228KB on datacenter SM100) or different instruction scheduling — then switching to TensorRT-LLM's implementation might bypass the problematic code. The trtllm backend uses NVIDIA's proprietary library, which may have different kernel implementations, different precision handling, or different memory access patterns that happen to work on SM120 where the open-source flashmla kernels fail.

There was also a clue in the server logs: the flashinfer_trtllm path was documented as SM100-only in the SGLang source code. But the trtllm NSA backend (as opposed to the flashinfer_trtllm MoE runner) might have broader hardware support. The assistant was essentially probing the boundary between what SGLang's developers had tested and what might work in practice.

Assumptions Embedded in This Message

Several assumptions are baked into this brief message. First, the assistant assumes that the server's "fired up and ready to roll!" message is trustworthy — that the warmup prefill completing without error is a meaningful signal. In practice, this was a reasonable assumption but not guaranteed: the NaN crashes in previous attempts had occurred during decode, not during prefill or warmup. The warmup phase exercises the model but may not trigger the same numerical pathways as extended autoregressive generation.

Second, the assistant assumes that the trtllm backend is actually available and functional on this system. The SGLang build at ~/sglang/ was compiled from source with the main branch, and TensorRT-LLM integration requires specific CUDA libraries and build flags. If the trtllm backend wasn't properly compiled or linked, the server might silently fall back to another backend or crash in a different way.

Third, there is an implicit assumption that the NaN issue is indeed backend-specific rather than a fundamental model compatibility problem. It's possible that the GLM-5-NVFP4 model has an inherent numerical instability on SM120 hardware regardless of attention backend — perhaps related to the NVFP4 quantization format, the DeepGemm scale format warning, or the transformers 5.2.0 RoPE incompatibility that SGLang itself warns about. The assistant is testing the hypothesis that the attention backend is the root cause, but other factors could independently cause NaN.

The Thinking Process Visible in the Todo List

The structured todo list embedded in the message reveals a disciplined, contingency-aware thought process. The assistant has organized the work into three priority levels with clear branching:

  1. Test Attempt 7 (high priority, in progress) — the immediate next action
  2. Tune sglang params for parallel query throughput (high priority, pending) — only relevant if Attempt 7 succeeds
  3. Run load tests with sglang.bench_serving (high priority, pending) — also contingent on success
  4. If decode fails: try --kv-cache-dtype bfloat16 — the fallback plan This structure shows that the assistant is thinking several steps ahead, preparing for both success and failure. The todo list is not just a record of what needs to be done; it's a decision tree encoded as task dependencies. The "if decode fails" branch acknowledges that this test might not work, and the next fallback — forcing bfloat16 KV cache — represents a different axis of exploration. If the NaN is caused by fp8 numerical issues in the KV cache path rather than the attention backend itself, switching to bfloat16 might resolve it at the cost of higher memory usage.

Input Knowledge Required to Understand This Message

To fully grasp what's happening in [msg 214], one needs substantial context about the deployment environment and the model. The key pieces of input knowledge include:

Output Knowledge Created by This Message

At the moment this message was written, the output knowledge was minimal — the assistant had confirmed the server was running but had not yet executed the critical test. However, the message itself creates important knowledge by explicitly documenting the state transition from "server loaded" to "about to test inference." This documentation serves several purposes:

  1. It establishes a clear checkpoint in the debugging narrative. Future readers (including the assistant itself in subsequent reasoning) can see exactly what configuration was being tested and what the expected outcome would be.
  2. It formalizes the success criteria. The todo list makes explicit that "decode works" is the gate for all subsequent tuning and benchmarking work.
  3. It preserves the contingency plan. By writing down the fallback options, the assistant ensures that if the test fails, the next steps are already defined and don't require starting from scratch. The message also creates implicit knowledge about the debugging methodology: the assistant is following a systematic, hypothesis-driven approach, testing one variable at a time (the NSA backend), and documenting each attempt with enough detail to compare results.

Mistakes and Incorrect Assumptions

While the message itself doesn't contain factual errors, several assumptions embedded in the broader context could be questioned. The most significant is the assumption that the NaN crash is caused by the attention backend specifically. The SGLang warning about transformers 5.2.0 RoPE incompatibility is a strong alternative candidate — if the rotary position embeddings are computed incorrectly, the attention scores could become numerically unstable regardless of which backend processes them. The assistant had noted this warning but prioritized the attention backend hypothesis, perhaps because the NaN manifests during decode (attention-heavy) rather than during prefill.

Another potential blind spot is the DeepGemm scale format warning. Even with --fp8-gemm-backend cutlass, DeepGemm might still be used for certain operations within the NSA attention pathway. The warning explicitly states "This might cause accuracy degradation on Blackwell" for non-ue8m0 scale formats. If the model checkpoint uses a different scale format, the numerical values flowing through the attention computation could be corrupted before they even reach the backend kernels.

The assistant also assumes that a successful single inference request would validate the configuration. In practice, the NaN crash might be intermittent — triggered only under certain memory pressure conditions, sequence lengths, or batch sizes. A single successful test would be encouraging but not definitive proof that the configuration is stable.

The Broader Significance

Message [msg 214] represents a turning point in a larger narrative about deploying state-of-the-art AI models on non-datacenter hardware. The GLM-5-NVFP4 model, with its 744B parameters and 256 experts, was designed for datacenter Blackwell GPUs with NVLink and abundant shared memory. Running it on eight PCIe-connected RTX PRO 6000 cards in a virtualized Proxmox environment is a fundamentally different proposition — one that requires finding workarounds for every assumption the model's developers made about the hardware.

The debugging journey captured in this session is emblematic of a broader trend in AI infrastructure: as models grow larger and more specialized, deploying them on anything other than the exact hardware they were designed for requires deep understanding of both the model architecture and the GPU programming stack. The assistant's systematic approach — documenting each attempt, reasoning about root causes, and maintaining contingency plans — is a model for how to approach such challenges.

Conclusion

Message [msg 214] is a moment of suspended anticipation in a debugging saga. The assistant has done the work: identified the likely culprit (NSA backends on SM120), selected a new variable to test (trtllm), launched the server with the new configuration, and verified that it loads and warms up successfully. All that remains is to send the actual inference request and see if the NaN crash returns.

The message's power lies in its honesty about the gap between "server started" and "inference works" — a distinction that is easy to overlook but critical in production ML deployments. The todo list structure reveals a disciplined, contingency-aware mindset, while the brevity of the prose belies the complexity of the debugging that preceded it. Whether Attempt 7 succeeds or fails, this message captures the methodology that makes systematic debugging possible: document the state, define the test, and prepare for both outcomes.