The Moment of Reckoning: A Systematic Debugging Chronicle for GLM-5-NVFP4 on Blackwell SM120 GPUs

In any complex engineering endeavor, there comes a moment when the accumulation of partial successes, dead ends, and half-understood failures must be crystallized into a coherent picture. Message 211 in this opencode conversation represents exactly such a moment — a comprehensive status report that is far more than a simple update. It is a knowledge artifact, a debugging diary, a strategic planning document, and a testament to systematic problem-solving in the face of one of the most challenging deployment scenarios in modern machine learning: running a 744-billion-parameter Mixture-of-Experts model with novel quantization and attention architectures across eight consumer-grade Blackwell GPUs connected only by PCIe.

This article examines message 211 in depth, exploring why it was written, what decisions it documents, the assumptions that shaped its analysis, and the rich tapestry of input and output knowledge it represents. The message stands at a critical juncture in the conversation — after approximately 210 prior exchanges spanning environment setup, driver installation, CUDA toolkit configuration, flash-attn compilation, and multiple failed server launches, the assistant pauses to take stock. The result is a document that any engineer debugging a similarly complex ML deployment would recognize as both familiar and invaluable.

The Anatomy of a Status Report

Message 211 is structured with the precision of a well-organized engineering notebook. It opens with a clear "Goal" section that restates the mission: deploy the GLM-5-NVFP4 model from HuggingFace (specifically the lukealonso/GLM-5-NVFP4 repository) on a remote machine with 8x NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang for inference serving, then tune for throughput and run load tests. This restatement is important — it re-anchors the reader (and the assistant itself) in the original objective after dozens of messages that could easily have led the conversation astray.

The message then proceeds through a carefully ordered series of sections: Instructions (operational notes for the deployment environment), Discoveries (hardware, software, and model details), Critical Issues Encountered (the NaN decode crash that has become the central blocker), Configurations Tried (a systematic table of seven attempts), What the HuggingFace Model Card Recommends, Key Learnings from FINDINGS.md (prior research on the same hardware), Accomplished (a triage of completed, in-progress, and not-yet-done items), Next Steps to Try if Attempt 7 Fails, and finally Relevant Files and Directories.

This structure reveals the assistant's thinking process. It is not merely reporting; it is organizing knowledge for future action. The message serves multiple audiences simultaneously: the human user who needs to understand the current state, the assistant itself as a reference for subsequent reasoning, and any future reader who might encounter similar problems. It is, in effect, a miniature knowledge base.

Why This Message Was Written

To understand why message 211 exists, we must consider the state of the conversation at this point. The assistant has been working through a multi-stage deployment process spanning three segments (as documented in the analyzer summaries). Segment 0 covered the full environment setup — NVIDIA drivers, CUDA toolkits, Python virtual environment, flash-attn compilation with its memory-exhaustion issues, and the initial SGLang installation. Segment 1 shifted to deployment, where the assistant encountered persistent NaN crashes during decode and began iterating through attention backends and quantization configurations. Segment 2, where this message resides, represents the culmination of those efforts.

By message 211, the assistant has tried seven different configurations. Each has failed with the same symptom: a CUDA device-side assert triggered by NaN or Inf values in the probability tensor during decode. The server loads successfully, the weights are distributed across eight GPUs, the KV cache is allocated, prefill warmup passes — but the moment actual generation begins, the model produces garbage.

This is a critical inflection point. The assistant has exhausted the obvious configuration permutations. It has read the local research repository's FINDINGS.md (an 871-line document documenting prior work with GLM-4.7, Kimi K2.5, and Kimi K2-Thinking on the same hardware). It has tried disabling CUDA graphs, switching FP8 GEMM backends, changing attention backends, and overriding NSA decode backends. Nothing has worked.

Message 211 is written because the assistant needs to consolidate everything it knows before proceeding to more radical interventions. The next steps section lists six options, including trying Docker images, investigating the transformers 5.2.0 RoPE incompatibility, filing a GitHub issue, and switching to the base FP8 model. These are not trivial changes — they represent significant shifts in strategy. Before undertaking them, the assistant must ensure that the knowledge gained from the seven failed attempts is not lost, that patterns can be discerned, and that the next attempt is informed by everything that has come before.

The Debugging Methodology: A Systematic Approach

The centerpiece of message 211 is the table of configurations tried. This table is a masterclass in systematic debugging. Each row represents a hypothesis about the root cause of the NaN crash, encoded as a specific combination of server parameters. Let us examine each attempt and the reasoning behind it.

Attempt 1: The initial configuration, essentially following the HuggingFace model card recommendations. It used flashinfer attention backend, auto-selected flashmla_kv for NSA decode, auto-selected DeepGemm for FP8 GEMM, fp8_e4m3 KV cache dtype, and CUDA graphs enabled. Result: OOM on CUDA graph capture. This revealed that the model card's recommended settings, designed for datacenter Blackwell (SM100) with 228KB of shared memory, were incompatible with the RTX PRO 6000's SM120 architecture which has only 101KB.

Attempt 2: Same configuration, but the OOM on graph capture was non-deterministic — it passed on retry. Result: NaN during decode. This was the first encounter with the core problem.

Attempt 3: Switched attention backend to triton. Result: AssertionError: nsa_kv_cache_store_fp8 incompatible. This revealed that the DSA (DeepSeek Sparse Attention) model architecture forces specific attention backends that are incompatible with triton.

Attempt 4: Changed NSA decode backend to flashmla_sparse. Result: NaN during decode. This eliminated the hypothesis that a specific NSA backend variant was the culprit.

Attempt 5: Changed FP8 GEMM backend to cutlass to avoid DeepGemm. Result: NaN during decode. This was a critical finding — it eliminated the DeepGemm scale format warning as the primary suspect. Even with DeepGemm disabled, the NaN persisted.

Attempt 6: Removed explicit KV cache dtype setting (letting it auto-select), disabled CUDA graphs. Result: NaN during decode. This eliminated both the KV cache dtype and CUDA graphs as root causes.

Attempt 7: Set both NSA decode and prefill backends to trtllm, kept cutlass for FP8 GEMM, disabled CUDA graphs. This was the current attempt at the time of writing — the server had just started and warmup had passed, but actual inference testing was pending.

This systematic elimination of variables is textbook debugging. Each attempt isolates one or two parameters, tests the hypothesis, and documents the result. The table format makes it easy to see which combinations have been tried and what the outcomes were. It also reveals the assistant's evolving understanding: the early attempts focused on DeepGemm and CUDA graphs, while later attempts shifted to NSA backends and KV cache configuration.

Assumptions and Their Consequences

Message 211 is rich with assumptions, some explicit and some implicit. Examining them reveals both the strengths and limitations of the assistant's approach.

Assumption 1: The NaN crash has a single root cause. The entire debugging effort assumes that one specific configuration parameter or combination is responsible for the numerical instability. This is a reasonable assumption, but it may be incorrect. The NaN could be caused by an interaction between multiple factors — for example, the combination of NVFP4 quantization with DSA attention on SM120 hardware might expose a bug that no single parameter change can fix. The next steps section implicitly acknowledges this by suggesting more radical alternatives like switching to the base FP8 model.

Assumption 2: The HuggingFace model card's recommendations are a reasonable starting point. The model card was written for datacenter Blackwell (SM100) GPUs, not the SM120 RTX PRO 6000. The assistant initially treated these recommendations as authoritative, only to discover that they led to OOM and NaN crashes. This assumption was corrected through experimentation, but it shaped the early attempts.

Assumption 3: The DeepGemm scale format warning is the primary suspect. The warning "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell" appears prominently in the server logs. It was natural to assume that this was the cause of the NaN. Attempt 5 disproved this by showing that even with --fp8-gemm-backend cutlass (which should avoid DeepGemm), the NaN persisted. However, the assistant notes that "DeepGemm may still be used for some ops" — a hedge that acknowledges the complexity of the software stack.

Assumption 4: NSA backends are interchangeable. The assistant tried flashmla_kv, flashmla_sparse, and trtllm as NSA decode backends, assuming that if one worked, the others would too. The fact that all produced NaN suggests either that the issue is upstream of the NSA backend selection, or that all available NSA backends share a common bug on SM120.

Assumption 5: The transformers 5.2.0 RoPE incompatibility is a secondary concern. The warning about RoPE parameter incompatibility is noted but deprioritized relative to the DeepGemm and NSA backend hypotheses. This may be a mistake — the RoPE issue could be the actual cause of the NaN, particularly if the model's positional encoding parameters are being computed incorrectly by the newer transformers version. The next steps section does include investigating this as option 2, suggesting the assistant recognizes it as a live possibility.

Assumption 6: The model weights themselves are correct. The assistant assumes that the GLM-5-NVFP4 checkpoint on HuggingFace is valid and that the NaN is caused by the serving stack, not the model. This is a reasonable assumption given that the model was presumably tested by its creators, but it is worth noting that the NVFP4 quantization format is novel and may have edge cases.

The Role of Prior Research

One of the most valuable aspects of message 211 is its integration of prior research from the local FINDINGS.md document. This 871-line file documented previous deployments of GLM-4.7 FP8, Kimi K2.5 INT4, and Kimi K2-Thinking NVFP4 on the same SM120 hardware. The assistant read this document (in messages 190-193) and extracted key insights:

Input Knowledge Required

To fully understand message 211, a reader needs substantial background knowledge spanning multiple domains:

Hardware Architecture: Understanding the difference between SM100 (datacenter Blackwell) and SM120 (RTX PRO 6000 Blackwell) is essential. The shared memory difference (228KB vs 101KB) directly explains why certain CUDA kernels fail. Knowledge of PCIe topology, NVLink, and GPU peer-to-peer communication helps contextualize the performance constraints.

Model Architecture: The GLM-5 model uses Mixture-of-Experts with 256 experts (8 activated per token), totaling 744B parameters with 40B active. It uses DeepSeek Sparse Attention (DSA), which is a variant of the attention mechanism that requires specialized NSA (Native Sparse Attention) backends. The NVFP4 quantization applies only to MoE expert MLPs, while attention layers remain in BF16.

Software Stack: The reader must understand the relationships between PyTorch, CUDA toolkits, flash-attn, flashinfer, SGLang, and transformers. The version compatibility matrix is complex: PyTorch 2.9.1 compiled against CUDA 12.8, SGLang from main branch (including PR #14311 for SM120 support), transformers 5.2.0 (required for the glm_moe_dsa model type), and flashinfer 0.6.3 with cubin support.

Debugging Methodology: The systematic approach of isolating variables, forming hypotheses, and testing them is a meta-skill that the message demonstrates but does not explain. A reader unfamiliar with this methodology might not appreciate the rigor behind the table of configurations.

SGLang Server Architecture: Understanding the various backends — attention backends (flashinfer, triton), NSA backends (flashmla_kv, flashmla_sparse, trtllm), FP8 GEMM backends (auto, cutlass, DeepGemm), MoE runner backends (flashinfer_cutlass, flashinfer_trtllm) — is necessary to interpret the configuration permutations.

Output Knowledge Created

Message 211 creates substantial new knowledge that extends beyond the immediate deployment task:

A Documented Failure Mode: The combination of GLM-5-NVFP4 with DSA attention on SM120 GPUs produces NaN during decode. This is a novel finding that may be valuable to the SGLang development team, the model creators, and anyone attempting similar deployments. The systematic documentation of seven failed attempts provides a roadmap of what does not work.

A Prioritized Action Plan: The next steps section transforms the debugging effort from reactive trial-and-error to a strategic plan. Each option is ranked and described with sufficient context for execution. This plan can guide the next phase of work without losing the insights from the failed attempts.

A Knowledge Base for Future Deployments: The message serves as a reference document for deploying large language models on non-datacenter hardware. The hardware topology notes, software version requirements, and configuration options are all documented in one place.

A Diagnostic Framework: The table of configurations tried establishes a methodology for diagnosing similar issues. Future debuggers can follow the same pattern: start with the recommended configuration, isolate variables systematically, document each attempt, and use the accumulated evidence to constrain the hypothesis space.

Cross-Referencing of Prior Work: By integrating findings from the local FINDINGS.md document, the message creates a bridge between previous deployments and the current challenge. This cross-referencing is itself a form of knowledge creation — it identifies the DSA attention as the novel element that distinguishes GLM-5 from previously successful NVFP4 deployments.

The Thinking Process Revealed

Message 211 reveals the assistant's thinking process through its structure, its choices of what to include and exclude, and its framing of the problem. Several aspects are particularly revealing:

The shift from execution to reflection: Earlier messages in the conversation are action-oriented — they issue bash commands, read files, and test configurations. Message 211 is reflective. It pauses the execution loop to synthesize. This shift is a hallmark of effective problem-solving: the recognition that continued action without reflection leads to diminishing returns.

The use of tables for pattern recognition: The configuration table is not just a record of attempts; it is a tool for pattern recognition. By arranging the data in a structured format, the assistant (and the human reader) can visually scan for patterns. Do all NaN crashes share a common parameter? Are there any configurations that avoided the NaN? The table makes these questions answerable.

The explicit acknowledgment of uncertainty: The message is careful to hedge its conclusions. "DeepGemm may still be used for some ops" acknowledges the limits of the assistant's understanding of the software stack. "Attempt 7 running... need to test actual inference" acknowledges that the current attempt is not yet validated. This epistemic humility is appropriate for a complex debugging scenario where certainty is elusive.

The prioritization of next steps: The six next steps are ordered by apparent likelihood of success and ease of implementation. Trying Docker images (option 1) is relatively easy and may provide a working environment. Investigating the RoPE issue (option 2) requires deeper analysis but addresses a specific warning. Filing a GitHub issue (option 4) is a fallback that acknowledges the possibility that the problem requires changes to SGLang itself.

The recognition of hardware constraints: The message repeatedly returns to the hardware differences between SM100 and SM120. This suggests that the assistant has internalized the hardware topology as a primary constraint and is using it to filter hypotheses. Any proposed solution that ignores the shared memory limitation or the PCIe-only interconnect is likely to be rejected.

Conclusion

Message 211 is far more than a status update. It is a carefully constructed knowledge artifact that serves multiple functions simultaneously: a debugging diary, a strategic planning document, a knowledge base for future deployments, and a moment of reflection in an otherwise action-oriented conversation. Its systematic documentation of seven failed attempts, its integration of prior research, its explicit acknowledgment of assumptions and uncertainties, and its prioritized plan for next steps all contribute to a model of effective engineering communication.

The message captures a critical inflection point in the deployment effort. The obvious configuration permutations have been exhausted. The assistant has eliminated several plausible hypotheses (DeepGemm, CUDA graphs, NSA backend selection) and is preparing to pursue more radical alternatives. The structured format ensures that the knowledge gained from the failed attempts is preserved and can inform future work, whether by the same assistant, a human engineer, or someone encountering similar challenges with different hardware or models.

For anyone debugging complex ML deployments, message 211 offers a template for how to pause, synthesize, and plan. It demonstrates that the most valuable output of a debugging session is not always the fix itself, but the structured understanding of what has been tried, what has been learned, and what remains to be investigated. In the high-stakes world of deploying state-of-the-art language models on non-standard hardware, such structured understanding is the difference between thrashing and progress.