When NaN Happens: Diagnosing FP4 Kernel Failures on Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, few moments are as simultaneously exciting and deflating as the first inference attempt on a freshly deployed model. After hours of building software stacks, patching source code, downloading multi-hundred-gigabyte checkpoints, and configuring systemd services, the moment of truth arrives — and the model responds with nothing but exclamation points. This is precisely the scenario captured in message 5843 of an extensive opencode coding session, where an AI assistant deploying the Qwen3.5-397B-A17B-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs discovers that its freshly deployed model is producing numerical garbage.

The Message

The subject message (msg id=5843) reads in full:

[assistant] That's garbage output — just ! repeated. The reasoning parser captured it as reasoning_content but the model is producing nonsense. This could be a quantization/kernel issue. Let me try without the reasoning parser and with higher temperature to see if it's a model loading issue: [bash] ssh root@10.1.230.174 'curl -s http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"qwen3.5-397b\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello, what model are you?\"}], \"max_tokens\": 100, \"temperature\": 0.7}" | python3 -m json.tool | head -20' { "id": "7b65139bbfbb4c818af701e8413de1c4", "object": "chat.completion", "created": 1772893618, "model": "qwen3.5-397b", "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "reasoning_content": null, "tool_calls": null }, "logprobs": null, "finish_reason": "stop", "matched_stop": "NaN happened" } ...

The message is deceptively short — just two curl commands and their results — but it represents a critical diagnostic pivot point. The assistant had just finished a multi-hour effort to deploy the Qwen3.5-397B-A17B-NVFP4 model, a 397-billion-parameter mixture-of-experts model quantized with NVIDIA's NVFP4 format, on a machine with eight Blackwell-generation GPUs connected via PCIe (without NVLink). The model had loaded successfully in about two minutes, the health endpoint returned 200, and the first test prompt — "Write a Python function to check if a number is prime. Be concise." — had produced an answer consisting entirely of exclamation marks, captured as reasoning_content by the Qwen3 reasoning parser.

The Diagnostic Process

The assistant's reasoning in this message reveals a methodical diagnostic approach. The first observation is that the output is "garbage" — specifically, repeated ! characters. This is not a subtle quality issue; it is catastrophic failure. The assistant immediately identifies two possible categories of root cause: a "quantization/kernel issue" or a "model loading issue."

The distinction is important. A model loading issue would mean the weights were corrupted during download or incorrectly loaded into memory — a problem that would require re-downloading or debugging the loader. A quantization/kernel issue would mean the numerical computations themselves are producing incorrect results, likely because the FP4 (4-bit floating point) kernels being used are incompatible with the Blackwell GPU architecture (compute capability 12.0, or SM120).

To distinguish between these, the assistant runs a second test with two key changes: it removes the reasoning parser (--reasoning-parser qwen3 was in the server args but the curl request doesn't use it) and increases the temperature from 0 to 0.7. The reasoning is that if the model were producing valid tokens that were simply being mis-parsed by the reasoning extractor, a different prompt without reasoning expectations might show meaningful content. And if the model were stuck in a degenerate mode at temperature 0 (always picking the highest-probability token, which might be !), higher temperature would introduce randomness and potentially break out of the loop.

The second test's result is even more revealing: "finish_reason": "stop" with "matched_stop": "NaN happened". This is a smoking gun. The SGLang inference engine explicitly detected that a NaN (Not a Number) value was produced during generation and terminated the sequence. NaN values in floating-point computation are a classic symptom of numerical instability — the model's weights and the input activations are being combined in a way that produces values outside the representable range, or the kernel itself has a bug that produces invalid floating-point values under certain conditions.## Context and Backstory

To fully understand this message, one must appreciate the journey that led to it. The session had been running for hours across multiple segments, tackling a series of increasingly complex deployment challenges. The machine featured eight NVIDIA RTX PRO 6000 Blackwell GPUs — cutting-edge hardware that was simultaneously the source of the system's power and its greatest compatibility challenge. Blackwell (compute capability 12.0, or SM120) was so new that many software components lacked proper support.

The assistant had already resolved numerous SM120-specific issues: patching SGLang's all-reduce utilities to recognize compute capability 12, enabling FlashInfer allreduce fusion for Blackwell, configuring Torch symmetric memory, and fixing the attention backend selection for hybrid GDN models (which combine linear attention and full attention layers). The Qwen3.5 model itself was a complex beast — 397 billion parameters with 512 experts (10 active per token), 60 layers, and a massive 262K-token context window, all compressed into 223 GB via NVFP4 quantization.

The first attempt to start the server had failed with a clear error: hybrid GDN models on Blackwell require triton or trtllm_mha attention backends, not flashinfer. The assistant fixed this, restarted, and the model loaded successfully in about two minutes. The health check passed. Everything looked good — until the first inference call produced nothing but exclamation marks.

Assumptions and Their Consequences

Several assumptions are visible in this message, some correct and some that would prove incorrect. The assistant's first assumption is that the garbage output might be a "reasoning parser" issue — that the model was producing valid content that was being incorrectly extracted as reasoning content. This was a reasonable hypothesis given that the first test's output showed reasoning_content filled with ! while content was null. However, the second test (which should have produced non-reasoning output) also failed, disproving this hypothesis.

The second assumption is that higher temperature might help the model escape a degenerate mode. This is a common debugging technique for language models — if a model is stuck outputting the same token at temperature 0, adding randomness can reveal whether the underlying probability distribution is reasonable. In this case, the NaN detection proved the problem was more fundamental: the model wasn't stuck in a local minimum of its probability distribution; it was producing mathematically invalid values.

The assistant also implicitly assumes that the model weights loaded correctly — that the download and loading process was successful. The fact that the server started, the health endpoint returned 200, and the model loaded in a reasonable time (2 minutes for 223 GB) all supported this assumption. The NaN output would eventually force a reconsideration of this assumption, but at this point in the diagnostic process, the assistant is still gathering data.

Input Knowledge Required

Understanding this message requires substantial domain knowledge. The reader must understand what FP4 quantization is — a 4-bit floating-point format that compresses model weights to reduce memory usage while attempting to preserve accuracy. They must understand that different GPU architectures (like Blackwell's SM120) may require specific kernel implementations for these quantization formats, and that using the wrong kernel can produce numerical errors. The concept of NaN in floating-point computation — a special value representing an undefined or unrepresentable result — is critical, as is understanding that "matched_stop": "NaN happened" is SGLang's explicit detection of this condition.

Knowledge of the SGLang serving framework is also necessary: the distinction between reasoning_content and content in the API response, the role of reasoning parsers and tool-call parsers, and the significance of finish_reason: "stop" versus other termination conditions. The reader must also understand the hardware context — eight Blackwell GPUs connected via PCIe without NVLink, the CUDA 13 software stack, and the fact that SM120 support was still being actively patched into the codebase.

Output Knowledge Created

This message creates critical diagnostic knowledge. It establishes definitively that the Qwen3.5-397B-A17B-NVFP4 model, when loaded with the current configuration on SM120 hardware, produces NaN values during inference. This is a hard failure — not a performance issue or a quality degradation, but a complete numerical breakdown. The "NaN happened" stop reason is particularly valuable because it narrows the possible causes: NaN in floating-point neural network inference typically arises from one of a few specific sources, including incorrect quantization kernels, incompatible matrix multiplication implementations, or numerical overflow in attention computations.

The message also creates negative knowledge — it rules out several potential causes. The problem is not a reasoning parser issue (the second test without reasoning expectations also failed). It is not a temperature-0 degeneracy issue (temperature 0.7 also failed). And it is not a subtle quality degradation that might be acceptable for some use cases — the model is producing mathematically invalid output that the inference engine itself recognizes as erroneous.

The Thinking Process

The assistant's thinking process, visible in the message's structure, follows a classic diagnostic pattern: observe the symptom, hypothesize the category of cause, design a test to distinguish between categories, and refine based on results. The first test established the symptom (garbage output). The assistant then hypothesized two categories: "quantization/kernel issue" versus "model loading issue." The second test was designed to gather more information by varying the prompt and temperature.

The decision to try "without the reasoning parser" is particularly insightful — it shows the assistant considering that the problem might be in the output processing pipeline rather than in the model itself. The reasoning parser was configured as qwen3 in the server args, and if it were incorrectly extracting tokens, it could produce the observed symptom. The second test's result — reasoning_content: null and content: null with NaN happened — definitively rules out this hypothesis.

The assistant's choice of a simple prompt ("Hello, what model are you?") is also strategic. When debugging a model that's producing garbage, the simplest possible prompt reduces the chance that the problem is prompt-specific. A model that can't answer "what model are you?" has a fundamental problem, not a prompt-engineering issue.

What Follows

The message ends with the assistant gathering evidence but not yet reaching a conclusion. The next steps would involve examining the server logs for FP4-related warnings, checking which FP4 GEMM runner backend was selected, and ultimately discovering — through a community reference (catid's GitHub gist) — that the default auto MoE runner backend and FP4 GEMM runner backend produce NaN on SM120. The fix would require explicitly setting --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn, and potentially rebuilding sgl-kernel from source with SM120 support.

This message captures the moment when a deployment that seemed successful — the server was running, the model was loaded, the health check passed — reveals itself to be fundamentally broken. It is the diagnostic pivot point that transforms a deployment task into a debugging mission, and it demonstrates the kind of systematic reasoning that separates effective troubleshooting from random experimentation.