The Breakthrough: GLM-5-NVFP4 Comes Alive on SM120 Blackwell

In the course of deploying a massive 744-billion-parameter mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message marks the inflection point where days of debugging, failed configurations, and systematic experimentation finally yielded success. Message [msg 217] is the assistant's triumphant confirmation that the GLM-5-NVFP4 model is "fully working on SM120!" — a moment that transforms a frustrating debugging saga into a working deployment.

The Context: A Seven-Attempt Debugging Ordeal

To understand the significance of this message, one must appreciate what preceded it. The assistant had been attempting to deploy the GLM-5-NVFP4 model — a 744B-parameter MoE model with 256 experts, 8 activated per token, quantized with NVFP4 (a 4-bit floating-point format) — on a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. These GPUs use the SM120 architecture, which is the workstation variant of Blackwell, distinct from the datacenter SM100 chips. The SM120 has only 101KB of shared memory compared to 228KB on SM100, a limitation that had already caused problems with attention kernels earlier in the session.

The model uses DeepSeek Sparse Attention (DSA), which forces the use of NSA (NVIDIA Sparse Attention) backends for both prefill and decode. This was the crux of the problem. Six prior attempts had all ended in failure, each crashing during the decode phase with a catastrophic error:

Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
torch.AcceleratorError: CUDA error: device-side assert triggered

The assistant had systematically tried different combinations of attention backends, NSA decode backends, FP8 GEMM backends, KV cache dtypes, and CUDA graph settings. Each row in the configuration table told the same story: flashmla_kv produced NaN, flashmla_sparse produced NaN, CUDA graphs caused OOM, the triton backend was incompatible. The model card's recommended configuration — designed for datacenter SM100 Blackwell — was useless on the SM120 hardware.

The Message Itself: Confirmation and Consolidation

Message [msg 217] opens with a bold declaration in bold text: "GLM-5-NVFP4 is fully working on SM120!" This is not a tentative observation but a definitive statement, backed by concrete evidence from two successful inference tests conducted in the preceding messages ([msg 215] and [msg 216]).

The assistant enumerates three key facts that collectively prove the model is functioning correctly:

  1. The model reasons through the problem — the reasoning_content field in the API response contains a coherent thinking chain, demonstrating that the model's internal reasoning mechanism is intact.
  2. It produces the correct answer — for the simple question "What is 2+2? Answer briefly," the model outputs &#34;4&#34; in the content field, which is objectively correct.
  3. It completes naturally — the finish_reason is &#34;stop&#34; rather than &#34;length&#34; or an error, meaning the model reached its natural conclusion without being truncated or crashing. These three points together constitute a complete validation: the model loads, it processes input, it generates coherent reasoning, it produces correct output, and it terminates cleanly. Every prior failure mode — NaN crashes, CUDA device-side asserts, OOM during graph capture — is absent.

The Deeper Test: Proving Robustness

The assistant does not stop at the simple 2+2 test. Recognizing that a single easy query might not reveal subtle instabilities, it immediately launches a more demanding evaluation: asking the model to "Write a Python function that checks if a number is prime. Include docstring and type hints." with 1024 max tokens. This tests the model's ability to sustain coherent generation over a longer sequence, to produce structured code output, and to follow formatting instructions.

The response confirms success. The model generates a complete Python function with proper imports (import math), type hints (n: int -&gt; bool), a docstring, and correct algorithmic logic. The truncated output shown in the message is enough to establish that the model is producing valid, executable code — not gibberish, not NaN-infested garbage, but real content.

The assistant then performs a final sanity check by examining the server log with tail -20. The log shows clean startup messages, no errors, no warnings about NaN or device-side asserts. The server reports max_total_num_tokens=370752, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=64, context_len=202752, and available_gpu_mem=17.08 GB — all normal operational metrics. The last line shows a healthy HTTP request being processed: 127....

The Reasoning Behind the Solution

The critical insight that led to this breakthrough was the selection of the NSA backends. In Attempt 7 ([msg 207]), the assistant launched the server with --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm, explicitly overriding the auto-selected backends that had failed in all prior attempts. The auto-selection mechanism, triggered by the model's forced kv_cache_dtype=&#39;fp8_e4m3&#39;, had been choosing flashmla_kv for decode and flashmla_auto for prefill — both of which crashed on SM120.

The trtllm backend, which stands for TensorRT-LLM, is a different implementation path that apparently handles the SM120 architecture correctly. The assistant's earlier analysis in [msg 211] had identified three potential root causes for the NaN crashes: DeepGemm scale format incompatibility, Transformers 5.2.0 RoPE parameter issues, and NSA attention backend incompatibility with SM120. By switching to trtllm, the assistant effectively eliminated the NSA backend hypothesis as the primary culprit, though it remains possible that the other factors were also contributing and were mitigated by the backend change.

Assumptions and Their Validation

Several assumptions underpin this message. The assistant assumes that the two successful test queries are representative — that the model will continue to work for arbitrary inputs, not just simple math and code generation. It assumes that the trtllm backend is stable under load, not just for single sequential requests. It assumes that the absence of errors in the server log tail means the system is healthy overall. These are reasonable assumptions for a validation checkpoint, but they are not yet proven — the subsequent tuning and load-testing phases will test them.

The assistant also implicitly assumes that the model's output quality is acceptable. The prime function test shows syntactically valid Python, but the full output is truncated in the message; we cannot see whether the function is logically complete and correct. The reasoning content from the 2+2 test shows the model working through the problem step by step, which is a strong indicator of quality.

Input Knowledge Required

To fully grasp this message, a reader needs to understand several layers of technical context. First, the hardware architecture: SM120 vs SM100, the shared memory limitation (101KB vs 228KB), and the implications for attention kernel execution. Second, the model architecture: what a 744B MoE model with 256 experts looks like, what NVFP4 quantization means, and why DSA forces NSA backends. Third, the software stack: sglang's server architecture, the role of NSA backends (flashmla_kv, flashmla_sparse, trtllm), and the interaction between KV cache dtype and backend selection. Fourth, the debugging methodology: the systematic configuration table, the hypothesis testing approach, and the interpretation of NaN/Inf errors in CUDA execution.

Output Knowledge Created

This message creates several important outputs. Most immediately, it establishes that GLM-5-NVFP4 can run on SM120 GPUs with the trtllm NSA backend — a finding that contradicts the implicit assumption that the model card's SM100-focused configuration was the only valid path. It validates the assistant's debugging methodology: systematic configuration enumeration, isolation of variables, and persistence through six failures. It also creates a validated configuration baseline that can be used for subsequent performance tuning: --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --fp8-gemm-backend cutlass --disable-cuda-graph --attention-backend flashinfer --moe-runner-backend flashinfer_cutlass.

Perhaps most importantly, this message serves as a forcing function for the next phase of work. The todo list from [msg 214] had two branches: "If decode works: tune sglang params" and "If decode fails: try alternative approaches." Message [msg 217] definitively selects the first branch, enabling the assistant to proceed with benchmarking, parameter tuning, and load testing — the original goals of the deployment task.

The Thinking Process Visible in the Message

The structure of the message reveals the assistant's thinking process. It opens with a bold summary of the key validation points, then immediately moves to further testing, then checks system health. This is classic scientific confirmation behavior: first announce the result, then attempt to replicate it with a harder test, then verify that the system is stable. The progression from "it works for a trivial case" to "it works for a non-trivial case" to "the server log shows no errors" demonstrates a methodical approach to validation that guards against false positives.

The message also shows the assistant's awareness of the narrative arc. The bold declaration "fully working on SM120!" is not just information — it is a milestone marker, a signal to both the user and the assistant itself that a significant barrier has been overcome. The assistant is not merely reporting results; it is framing them within the larger story of the deployment effort.

Conclusion

Message [msg 217] represents the climax of a multi-attempt debugging saga. After six failed configurations, each crashing with NaN errors during decode, the seventh attempt using trtllm NSA backends finally produces coherent, correct output from the GLM-5-NVFP4 model on SM120 Blackwell GPUs. The message is simultaneously a validation report, a robustness test, and a handoff to the next phase of work. It demonstrates the value of systematic hypothesis testing, the importance of understanding hardware-specific backend behavior, and the persistence required to deploy cutting-edge models on non-standard hardware configurations.