The Decode That Broke: Diagnosing a CUDA Kernel Crash on Blackwell GPUs

In the high-stakes world of deploying cutting-edge AI models on novel hardware, a single error message can redirect hours of effort. Message 166 of this opencode session captures precisely such a moment: the assistant, having successfully loaded the massive GLM-5-NVFP4 model (a 400GB-quantized Mixture-of-Experts model) across eight NVIDIA RTX PRO 6000 Blackwell GPUs, watches the server crash during its very first inference request. The message is a masterclass in diagnostic narrowing—taking a vague "device-side assert triggered" CUDA error and beginning the process of isolating its root cause.

The Context: A Long Road to a Crash

To understand message 166, one must appreciate the journey that preceded it. The session began in segment 0 with the complete setup of an ML environment on Ubuntu 24.04: NVIDIA driver installation, CUDA Toolkit 13.1, Python virtual environment management with uv, and the notoriously difficult compilation of flash-attn on the new SM120 Blackwell architecture. After resolving build memory exhaustion, version incompatibilities between PyTorch and vLLM, and a machine upgrade from 2 to 8 GPUs, the assistant finally turned to deploying the GLM-5-NVFP4 model using SGLang.

The deployment itself was a saga. The initial SGLang release (0.5.8.post1) lacked a critical SM120 shared memory fix, requiring a build from the main branch (PR #14311). The model's novel glm_moe_dsa architecture forced an upgrade to Transformers 5.2.0. The first server launch hit a CUDA out-of-memory error during graph capture because --mem-fraction-static 0.95 was too aggressive. After lowering it to 0.88, the server started successfully, KV cache was allocated (425,664 tokens across 8 GPUs), CUDA graphs were captured, and the server began accepting requests.

Then came the crash.

The Message: A Diagnostic Pivot

Message 166 opens with the assistant's sharpest diagnostic insight yet:

The error is during decode, not prefill (the prefill completed successfully per the log). This could be a flashinfer/flashMLA kernel issue on SM120.

This single sentence represents a critical narrowing of the hypothesis space. The assistant has examined the logs and noticed something subtle: the prefill phase (processing the input prompt) completed without error. The crash occurred during decode (generating tokens). This distinction is vital because prefill and decode use different computational paths—different CUDA kernels, different memory access patterns, different parallelism strategies. A crash that only manifests during decode points to a different class of problems than one that affects both phases.

The assistant then articulates a specific hypothesis: "flashinfer/flashMLA kernel issue on SM120." This is not a guess from thin air. SM120 is the compute architecture of the Blackwell GPU (RTX PRO 6000), and flashinfer is the attention backend that SGLang uses by default for this model. FlashMLA is the Multi-head Latent Attention implementation used for decode. The assistant is connecting the architecture-specific nature of the crash (SM120) with the specific kernel path (flashinfer's decode backend).

The Tool Call: Extracting Evidence

The assistant immediately follows its hypothesis with a targeted investigation:

ssh 10.1.230.175 'grep -i "nsa\|dsa\|attention.*backend\|flashmla\|decode.*backend\|triton.*backend" ~/sglang-glm5.log | head -20'

This grep command is carefully crafted. It searches for five patterns that each represent a potential clue:

WARNING model_config.py:1025: Transformers version 5.2.0 is used for model type glm_moe_dsa. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround.
WARNING server_args.py:1207: Force NSA prefill to use MLA (i.e. disable MHA_ONE_SHOT) for GlmMoeDsaForCausalLM on Blackwell.

These warnings are gold. The first warns that Transformers 5.2.0—which was required to support the glm_moe_dsa architecture—may have RoPE (Rotary Position Embedding) parameter incompatibilities. A RoPE miscalculation during decode could easily produce out-of-bounds indices or NaN values, triggering a device-side assert. The second warning reveals that SGLang is forcing NSA prefill to use MLA on Blackwell, which is a workaround that may itself have issues.

The Reasoning Process: What the Assistant Knows

Message 166 reveals several layers of the assistant's thinking:

Layer 1: Temporal reasoning. The assistant has read the server logs carefully enough to distinguish prefill from decode timing. This requires understanding SGLang's internal architecture—that prefill and decode are separate phases with separate kernel invocations.

Layer 2: Architecture-specific reasoning. The assistant knows that SM120 (Blackwell) is a new GPU architecture and that CUDA kernel compatibility is a known challenge. The mention of "flashinfer/flashMLA kernel issue on SM120" shows awareness that attention kernels often need architecture-specific tuning.

Layer 3: Log analysis methodology. The assistant doesn't just grep for "error"—it searches for specific configuration-related terms (NSA, DSA, attention backends) that might reveal why a particular kernel path was chosen and whether that path is known to be problematic.

Layer 4: Research awareness. The assistant mentions wanting to "look at the FINDINGS.md in this repo for any relevant notes from previous runs." This shows awareness that the local repository contains accumulated knowledge from previous debugging sessions on the same hardware—a form of institutional memory that could shortcut the debugging process.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message, some more justified than others:

Assumption 1: The crash is in an attention kernel. The assistant hypothesizes "flashinfer/flashMLA kernel issue." While plausible (attention is the most compute-intensive part of decode), the crash could equally be in the MoE routing, the quantization/dequantization kernels, or the sampling backend. The assistant is narrowing to attention as a first hypothesis, not a confirmed diagnosis.

Assumption 2: The RoPE warning is relevant. The Transformers version warning about RoPE parameters is treated as a potential clue. However, the warning is generic—it fires for any model using Transformers 5.2.0 with glm_moe_dsa. It may be a red herring, or it may be the exact cause. The assistant wisely doesn't commit to this hypothesis but files it as evidence.

Assumption 3: The prefill/decode distinction is meaningful. The assistant assumes that because prefill completed successfully, the problem is specific to decode. This is a reasonable inference, but CUDA device-side asserts can be asynchronous—the error could have originated during prefill but only been reported during the next CUDA API call (which happens to be in decode). The assistant's own grep output from a previous message acknowledges this: "CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect."

Assumption 4: SM120 compatibility is the root cause. The assistant frames this as a "kernel issue on SM120," implying the Blackwell architecture is the primary variable. While this is likely true—Blackwell is new and many CUDA kernels need updates—there could be other factors like the specific model architecture (DSA), the quantization format (NVFP4), or the tensor parallelism configuration.

Input Knowledge Required

To fully understand message 166, one needs:

  1. SGLang architecture knowledge: Understanding that SGLang separates prefill and decode into distinct scheduler phases, and that attention backends (flashinfer, triton, flashmla) are pluggable components.
  2. CUDA error semantics: Knowing that "device-side assert triggered" means a kernel executed on the GPU encountered an unexpected condition (like NaN, out-of-bounds index, or division by zero) and aborted.
  3. Blackwell/SM120 awareness: Understanding that SM120 is NVIDIA's newest GPU architecture (Blackwell generation) and that CUDA kernel compatibility is an ongoing challenge for early adopters.
  4. Model architecture familiarity: Knowing that GLM-5 uses a DeepSeek-style Attention (DSA) mechanism with MLA (Multi-head Latent Attention) and MoE (Mixture of Experts), and that these components each have separate CUDA kernel implementations.
  5. Quantization concepts: Understanding NVFP4 (NVIDIA FP4 quantization) and that quantized models require specialized dequantization kernels that may not be optimized for all architectures.
  6. The session's history: The earlier struggles with OOM, the main-branch SGLang build for the SM120 fix, and the Transformers version requirement.

Output Knowledge Created

Message 166 produces several valuable outputs:

  1. A confirmed diagnostic boundary: The crash is now localized to the decode phase, eliminating prefill, model loading, KV cache allocation, and CUDA graph capture as potential causes.
  2. Two concrete leads: The Transformers RoPE warning and the NSA/MLA backend selection for Blackwell are now on the table as potential causes.
  3. A research direction: The assistant has identified that it should consult FINDINGS.md for prior art on this specific hardware/software combination.
  4. A methodology for further investigation: The assistant has demonstrated how to extract relevant configuration warnings from the log, providing a template for subsequent debugging steps.
  5. A hypothesis to test: The flashinfer/flashMLA kernel compatibility on SM120 is now the primary theory to be confirmed or refuted in subsequent messages.

The Broader Significance

Message 166 represents a transition from "something is broken" to "let me understand exactly what is broken and why." This is the moment where debugging shifts from reactive (trying different configurations at random) to hypothesis-driven (forming theories and testing them). The assistant's ability to distinguish prefill from decode, to connect the crash to specific kernel paths, and to extract configuration warnings as evidence, all demonstrate a systematic approach to diagnosing novel hardware-software incompatibilities.

In the messages that follow (167–172), the assistant builds on this foundation. It kills the crashed server, tries the --attention-backend triton fallback, discovers that SGLang still overrides the attention backend selection for DSA models (forcing flashmla_kv for decode), and eventually consults the local FINDINGS.md which documents the DeepGemm scale format incompatibility as a known issue on Blackwell. But all of these subsequent steps depend on the diagnostic foundation laid in message 166: the recognition that the crash is decode-specific, attention-kernel-related, and architecture-dependent.

This message is a reminder that effective debugging of complex AI infrastructure is not about guessing—it's about methodically narrowing the hypothesis space until the root cause becomes visible. The assistant's disciplined approach to log analysis, its understanding of the system's internal architecture, and its ability to form and test specific hypotheses, all converge in this single, pivotal message.