The Pivot: Downgrading Transformers to Debug NaN Crashes in GLM-5-NVFP4

In the midst of a grueling debugging session deploying the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message marks a critical turning point. The assistant, having exhausted a series of configuration tweaks that all failed to resolve a persistent NaN crash during model decode, pivots to an entirely new hypothesis. Message 201 captures this moment of re-evaluation and the decisive action that follows.

The Message

Here is the message in full:

SGLang has native support for GlmMoeDsaForCausalLM. The config loading through get_config is what requires transformers to know the model type. Let me check if we can downgrade transformers but still make sglang load the config by using trust_remote_code:

>

[bash] ssh 10.1.230.175 '~/.local/bin/uv pip install --python ~/ml-env/bin/python3 "transformers==4.57.6" 2>&1 | tail -5'

>

Installed 2 packages in 104ms - huggingface-hub==1.4.1 + huggingface-hub==0.36.2 - transformers==5.2.0 + transformers==4.57.6

On the surface, this looks like a simple package downgrade. But beneath it lies a sophisticated chain of reasoning that re-frames the entire debugging effort.

The Context: A Cascade of Failed Fixes

To understand why this message matters, one must appreciate the dead end the assistant had reached. The deployment of GLM-5-NVFP4—a massive Mixture-of-Experts model quantized to FP4 precision—had been repeatedly crashing during the decode phase with a device-side assert triggered error. The root cause, as revealed by the error logs, was that the probability tensor contained inf, nan, or elements less than zero. This is a catastrophic numerical failure: when a language model's output distribution contains NaN values, sampling becomes impossible and the inference server crashes.

The assistant had tried everything. It switched attention backends from flashinfer to triton, only to hit an assertion that the triton backend doesn't support NSA (DeepSeek Sparse Attention) with FP8 KV cache. It tried flashmla_sparse as the NSA decode backend, but the crash persisted. It forced --fp8-gemm-backend cutlass to avoid the DeepGemm library, which had been emitting warnings about an incompatible checkpoint scale format (ue8m0). It disabled CUDA graphs. It tried trtllm backends. Every attempt ended the same way: the server would load successfully, warm up, and then crash on the very first real decode request.

The assistant had consulted the local research repository (FINDINGS.md), which documented successful deployments of a similar model (Kimi K2-Thinking NVFP4) on the exact same hardware. That model worked fine. The critical difference was that GLM-5 uses the glm_moe_dsa architecture, which employs DeepSeek Sparse Attention (DSA), and this architecture required upgrading the Transformers library to version 5.2.0.

The Insight: Tracing the NaN to Its Source

The assistant's reasoning in message 201 reveals a crucial realization. It states: "SGLang has native support for GlmMoeDsaForCausalLM. The config loading through get_config is what requires transformers to know the model type."

This is the key insight. The assistant had upgraded Transformers to 5.2.0 specifically because the GLM-5 model's glm_moe_dsa architecture was not recognized by older versions. But SGLang itself—the serving framework—has its own model implementations. The file glm4_moe.py in the SGLang source code registers GlmMoeDsaForCausalLM directly. The only reason Transformers 5.2.0 was needed was for the initial configuration loading step, where get_config parses the model's config.json and needs to know the architecture type.

But here's the critical detail: Transformers 5.2.0 itself had emitted a warning during model loading: "If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0." This warning, which had been noted earlier in the session, now became the prime suspect. If the upgraded Transformers library was introducing incompatible RoPE (Rotary Position Embedding) parameter handling, it could easily produce NaN values in the attention computation during decode.

The assistant's hypothesis, therefore, is that the NaN crash is not caused by the attention backend choice, nor by the DeepGemm scale format incompatibility, but by the Transformers 5.2.0 upgrade itself. The RoPE parameter incompatibility could silently corrupt the positional encodings, leading to numerical instability that only manifests during the decode phase when the model actually generates tokens.

The Decision: A Surgical Downgrade

The assistant's proposed solution is elegant and surgical: downgrade Transformers from 5.2.0 to 4.57.6, but rely on SGLang's native model support and the trust_remote_code flag to handle the model architecture. The trust_remote_code flag tells Hugging Face's loading infrastructure to execute the custom model code shipped with the model repository itself, bypassing the need for Transformers to have built-in knowledge of the architecture.

This approach has a clear logic: if SGLang already knows how to run GlmMoeDsaForCausalLM through its own model registry, and the model repository provides its own configuration class via trust_remote_code, then Transformers 4.57.6 should be sufficient. The only thing lost by downgrading is the built-in architecture registry that Transformers 5.2.0 provides—but that registry was only needed for the initial config parsing, and SGLang can handle that through alternative means.

The command issued is straightforward: uv pip install "transformers==4.57.6". The result shows a clean downgrade, with Transformers dropping from 5.2.0 to 4.57.6 and Hugging Face Hub regressing from 1.4.1 to 0.36.2 as a dependency adjustment.

Assumptions and Risks

This decision rests on several assumptions. First, the assistant assumes that the RoPE incompatibility warning from Transformers 5.2.0 is the actual cause of the NaN crash, rather than the DeepGemm scale format issue or some other SM120 kernel incompatibility. This is a reasonable inference given that all other configuration changes failed, but it is not proven.

Second, the assistant assumes that SGLang's native model support for GlmMoeDsaForCausalLM is complete and does not depend on any Transformers 5.x APIs internally. If SGLang's glm4_moe.py model implementation calls Transformers utility functions that changed between 4.57.6 and 5.2.0, the downgrade could introduce new errors.

Third, the assistant assumes that trust_remote_code will successfully load the model's custom configuration without requiring the newer Transformers version. This is plausible but not guaranteed—the model's custom config code might itself depend on Transformers 5.x internals.

There is also a subtle risk: the downgrade of huggingface-hub from 1.4.1 to 0.36.2 could affect model downloading or caching behavior. The Hugging Face Hub library handles authentication, caching, and file downloading. An older version might lack features or bug fixes, though for basic model loading this is unlikely to be problematic.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. The reader must understand what Transformers library versioning means in the ML ecosystem—that major version jumps (4.x to 5.x) often involve breaking API changes. One must know what RoPE (Rotary Position Embedding) is and why parameter incompatibility could cause numerical instability in attention computations. Knowledge of SGLang's architecture is helpful: the fact that it maintains its own model implementations separate from Transformers, and that it can register custom architectures like GlmMoeDsaForCausalLM without relying on Transformers' model registry. Finally, understanding the trust_remote_code mechanism in Hugging Face's loading pipeline—that it allows executing arbitrary Python code from the model repository to handle custom architectures—is essential.

Output Knowledge Created

This message creates several valuable pieces of knowledge. It establishes that SGLang has native support for the GlmMoeDsaForCausalLM architecture, which means the Transformers upgrade was only needed for config loading, not for runtime execution. It documents a specific hypothesis about the root cause of the NaN crash: that Transformers 5.2.0's RoPE parameter handling is incompatible with GLM-5's model weights. It also demonstrates a debugging methodology: when multiple configuration changes all fail to resolve an issue, step back and question the foundational assumptions—in this case, whether the Transformers upgrade was necessary at all.

The Thinking Process

The reasoning visible in this message shows a mature debugging approach. The assistant does not simply try another random configuration flag. Instead, it re-examines the dependency chain: Transformers 5.2.0 was introduced specifically for this model, it emitted a warning about RoPE compatibility, and the crash involves NaN values in the probability tensor—exactly the kind of numerical instability that incorrect position encodings would produce. The assistant connects these dots and formulates a testable hypothesis.

The phrase "Let me check if we can downgrade transformers but still make sglang load the config by using trust_remote_code" reveals the experimental mindset. The assistant is not certain this will work—it's a check, an exploration. The downgrade itself is executed quickly and cleanly, with the output showing a successful package swap in just 104 milliseconds.

Aftermath and Significance

Whether this downgrade ultimately resolves the NaN crash is a question answered in subsequent messages. But regardless of the outcome, message 201 represents a methodological shift in the debugging process. It moves from tweaking runtime parameters (attention backends, GEMM backends, CUDA graph settings) to questioning the software stack itself. This is the kind of lateral thinking that complex system debugging often requires: when the obvious knobs stop working, look at the foundations.

The message also illustrates a broader truth about deploying cutting-edge AI models on new hardware. The GLM-5-NVFP4 model, the Blackwell SM120 architecture, and the SGLang serving framework are all rapidly evolving. When these components interact in unexpected ways, the debugging process becomes a detective story—gathering clues from error messages, warnings, log files, and research notes, then forming and testing hypotheses until the culprit is found.