The DeepGemm Scale Format Trap: Debugging NaN Crashes in GLM-5-NVFP4 on Blackwell GPUs

Introduction

In any complex machine learning deployment, the most frustrating bugs are those that manifest as silent numerical corruption—where the system appears to initialize correctly, loads without errors, and then produces garbage output that triggers a crash only at the sampling stage. Message [msg 187] captures the precise moment in a debugging session where the assistant, after hours of iterating through false leads, finally identifies the root cause of persistent NaN crashes during inference of the GLM-5-NVFP4 model on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. This message represents a breakthrough: the recognition that an auto-selected GEMM backend—DeepGemm—was producing numerical garbage because its expected scale format (ue8m0) did not match the format stored in the model checkpoint. The solution was to explicitly override the FP8 GEMM backend to cutlass, a safer fallback that respects the checkpoint's actual quantization layout.

The Context: A Cascade of False Leads

To understand the significance of this message, one must appreciate the debugging journey that preceded it. The assistant had been attempting to deploy the GLM-5-NVFP4 model—a massive Mixture-of-Experts (MoE) language model with 8-bit floating-point (FP4) quantization—across 8 Blackwell GPUs using SGLang, a high-performance inference engine. The server would start successfully, load all 83 checkpoint shards, capture CUDA graphs, and even complete a prefill warmup. But the moment a decode step was attempted, the server would crash with a cryptic device-side assert triggered error, traced to a probability tensor contains either inf, nan or element < 0 assertion ([msg 185]).

The assistant had chased several plausible explanations. First, it suspected an SM120 (Blackwell architecture) incompatibility with the attention backend, trying triton and flashmla_sparse backends ([msg 170], [msg 178]). The triton backend failed with a different assertion (assert not self.nsa_kv_cache_store_fp8), confirming it was incompatible with the DSA/NSA sparse attention that GLM-5 uses. The flashmla_sparse backend loaded successfully but still crashed during decode with the same NaN error. Next, the assistant investigated a Transformers 5.2.0 warning about RoPE parameter incompatibilities ([msg 166]). It consulted the local research repository's FINDINGS.md, searched HuggingFace discussions, and even looked at GitHub issues for SM120 kernel problems ([msg 169]). Each lead was reasonable, but none addressed the actual cause.

The Breakthrough: Connecting the Warnings

The critical insight in [msg 187] comes from connecting two pieces of information that had been visible but not yet linked. Earlier in the logs, a warning had appeared: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell" ([msg 186]). The assistant had seen this warning but initially dismissed it as a minor accuracy concern rather than a crash-causing issue. The NaN crash—"probability tensor contains either inf, nan or element < 0"—was the direct consequence of this scale format mismatch.

The reasoning chain in the message is precise and economical. The assistant states: "DeepGemm auto-selects on Blackwell but the checkpoint's scale format is incompatible, causing NaN outputs." This sentence encapsulates the entire debugging breakthrough. DeepGemm is a high-performance GEMM (General Matrix Multiply) library optimized for Blackwell's tensor cores, and SGLang's fp8_gemm_runner_backend defaults to auto, which correctly identifies Blackwell hardware and selects DeepGemm as the optimal backend. However, DeepGemm expects weight scales stored in the ue8m0 format (unsigned 8-bit exponent, 0-bit mantissa), while the GLM-5-NVFP4 checkpoint uses a different scale format. When DeepGemm interprets the scales incorrectly, the matrix multiplication outputs become garbage—NaN or Inf values that propagate through the MoE layers and only surface when the sampling kernel attempts to create a probability distribution from the corrupted logits.

The Solution: Explicit Backend Override

The assistant's solution is to force the FP8 GEMM backend away from DeepGemm by explicitly setting --fp8-gemm-backend cutlass. The Cutlass backend, while potentially slower than DeepGemm on Blackwell hardware, correctly handles the checkpoint's actual scale format. The assistant also notes that the HuggingFace model card explicitly recommends --moe-runner-backend flashinfer_cutlass and --attention-backend flashinfer, but does not mention disabling DeepGemm—an oversight in the model card's documentation that led the assistant down several wrong paths.

The command issued in this message launches a new server with the critical --fp8-gemm-backend cutlass flag added, alongside the previously validated flags: tensor parallelism 8, FP4 quantization (modelopt_fp4), FP8 KV cache, flashinfer attention backend, and the flashinfer_cutlass MoE runner. The environment variables (CUDA_HOME, NCCL_*, OMP_NUM_THREADS) are carried over from previous successful configurations.

Input Knowledge Required

To understand this message, one needs knowledge of several domains. First, an understanding of the Blackwell GPU architecture (SM120) and its tensor core capabilities, including the ue8m0 scale format used by DeepGemm. Second, familiarity with SGLang's server arguments and how the fp8_gemm_runner_backend, moe-runner-backend, and attention-backend interact—specifically that auto selects DeepGemm on Blackwell. Third, knowledge of the GLM-5-NVFP4 model's quantization scheme and the HuggingFace model card's recommended deployment parameters. Fourth, an understanding of how NaN propagation works in neural network inference: a single corrupted matrix multiplication can produce Inf values that, when passed through softmax or sampling, trigger device-side assertions. Fifth, familiarity with the debugging workflow of inspecting server logs, identifying warning messages, and correlating them with crash signatures.

Output Knowledge Created

This message creates several important pieces of knowledge. First, it establishes that the GLM-5-NVFP4 checkpoint's scale format is incompatible with DeepGemm's ue8m0 expectation on Blackwell GPUs. Second, it documents a working workaround: forcing --fp8-gemm-backend cutlass to bypass DeepGemm. Third, it reveals a gap in the HuggingFace model card's documentation—it recommends flashinfer_cutlass for the MoE runner but does not warn users about the DeepGemm scale format issue. Fourth, it demonstrates that the NaN crash during decode is not an SM120 kernel bug, a RoPE parameter issue, or an attention backend incompatibility, but rather a numerical precision problem in the GEMM layer. Fifth, it provides a validated command template for deploying GLM-5-NVFP4 on Blackwell hardware that future users can reference.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message. It assumes that the Cutlass backend will correctly handle the checkpoint's scale format—a reasonable assumption given that Cutlass is a more general-purpose library that doesn't require the ue8m0 format. It assumes that the performance penalty of using Cutlass over DeepGemm is acceptable, though this is not explicitly discussed. It assumes that the other flags (--moe-runner-backend flashinfer_cutlass, --attention-backend flashinfer) are compatible with the Cutlass FP8 GEMM backend. It also assumes that the NaN issue is solely caused by the DeepGemm scale format mismatch and not by a combination of factors including the Transformers 5.2.0 RoPE issue mentioned in earlier warnings.

A potential mistake is that the assistant does not verify whether the Cutlass backend actually resolves the NaN issue before writing this message—the server is launched but the result is not yet known. The message ends with PID: 18476, indicating the process has started, but we don't see the outcome until subsequent messages. This is typical of iterative debugging: the hypothesis is formed, a test is launched, and results are checked in the next round.

The Thinking Process

The reasoning visible in this message is a model of diagnostic thinking. The assistant synthesizes three pieces of evidence: (1) the DeepGemm scale format warning from the logs, (2) the NaN/Inf probability tensor assertion, and (3) the model card's omission of a DeepGemm override. It recognizes that the auto default for fp8_gemm_runner_backend is the mechanism by which DeepGemm gets selected. It then formulates a minimal intervention—changing only the FP8 GEMM backend while keeping all other flags identical to the model card's recommendations. The thinking is linear and decisive: identify the root cause, propose the fix, and execute it. There is no hedging or exploration of alternatives; the assistant is confident enough in its diagnosis to commit to a single approach.

Conclusion

Message [msg 187] is a classic debugging breakthrough moment. After hours of chasing attention backends, RoPE parameters, and CUDA kernel compatibility issues, the assistant identifies the true culprit: an auto-selected GEMM backend whose scale format expectations don't match the model checkpoint. The solution is elegant in its simplicity—a single flag change that overrides the default backend selection. This message illustrates a crucial lesson in ML infrastructure: the most performant default is not always the correct default, and numerical correctness must take precedence over raw throughput. The DeepGemm scale format trap is a cautionary tale about the hidden assumptions baked into auto-detection logic, and the importance of reading warning messages—even those that sound like minor accuracy concerns—as potential crash causes.