Diagnosing NaN Outputs: The FP4 Kernel Detective Work on Blackwell GPUs

The Subject Message

[assistant] `NaN happened` — there's a numerical issue with the FP4 kernels. This is likely the 
`flashinfer_cutlass` FP4 GEMM runner not working properly on SM120. Let me check what backends 
are available and try the `sgl-kernel` one instead:
[bash] ssh root@10.1.230.174 'grep -n "fp4_gemm_runner_backend\|FP4.*backend\|cutlass.*fp4\|nvfp4" 
/root/sglang-main/python/sglang/srt/server_args.py | head -10'
107:    "petit_nvfp4",
252:def add_fp4_gemm_runner_backend_choices(choices):
469:    fp4_gemm_runner_backend: str = "flashinfer_cutlass"
607:    disable_flashinfer_cutlass_moe_fp4_allgather: bool = False
2220:        """Check FP4 KV cache compatibility with the attention backend"""
3398:            "Supported values: 'fp8', 'int4_awq', 'w4a8_awq', 'nvfp4', 'nvfp4_awq'. "
4240:            default=ServerArgs.fp4_gemm_runner_backend,
4241:            dest="fp4_gemm_runner_backend",
4895:           ...

Context: The Model That Could Only Say "!"

This message marks a pivotal diagnostic moment in a complex deployment effort. The assistant had just finished deploying the Kimi-K2.5 INT4 model with EAGLE-3 speculative decoding — a hard-won success after weeks of kernel debugging, CUDA version upgrades, and NCCL tuning. The user then pivoted to a newer, more efficient model: nvidia/Qwen3.5-397B-A17B-NVFP4, a 397-billion-parameter Mixture-of-Experts model quantized to NVIDIA's NVFP4 format, with only 17 billion active parameters per token thanks to its 512-expert, 10-experts-per-token architecture.

The initial deployment seemed straightforward. The model had been downloaded (223 GB across 6 safetensor shards), the latest SGLang main branch had been built from source, and SM120 patches had been applied to enable FlashInfer allreduce fusion and Torch symmetric memory on Blackwell. The server started successfully — but when the assistant sent a test prompt asking the model to write a Python function to check if a number is prime, the response was catastrophic: the model output nothing but repeated exclamation marks (!!!!!!!!!!!!!!!!...), captured as reasoning_content. A second test with a simple greeting resulted in finish_reason: "NaN happened".

The model was producing numerical garbage — and worse, the NaN finish reason indicated that the FP4 matrix multiplication kernels were generating floating-point values so corrupted that they could not even be sampled from.

Why This Message Was Written: The NaN Crisis

The message exists because a production deployment had just failed in a particularly baffling way. The assistant had invested significant effort into getting the Qwen3.5 model loaded — fixing the attention backend from flashinfer to triton after discovering that hybrid GDN models (which mix linear attention layers with full attention) require specific backends on Blackwell, waiting through the 2-minute weight loading process, and verifying that the server reported healthy status. Everything looked correct until the actual inference produced garbage.

The NaN happened finish reason is SGLang's way of signaling that the logits generated by the model contained NaN (Not a Number) values, which caused the sampling step to abort. This is a hard failure — it means the numerical computation somewhere in the model's forward pass produced invalid floating-point values. On Blackwell GPUs running NVFP4-quantized models, the most likely culprit is the FP4 GEMM (General Matrix Multiply) runner, which handles the 4-bit floating-point matrix multiplications that are the core computation of the quantized model.

The assistant's reasoning chain is visible in the first line: "there's a numerical issue with the FP4 kernels. This is likely the flashinfer_cutlass FP4 GEMM runner not working properly on SM120." This is a well-informed hypothesis. The assistant knows that:

1. SM120 (Blackwell) is a new compute architecture that requires specific kernel support
2. FP4 quantization relies on specialized GEMM runners that may not have been validated on SM120
3. The default backend (flashinfer_cutlass) was selected automatically by SGLang and may be incompatible

The Diagnostic Approach

The message shows a systematic diagnostic methodology. Rather than guessing randomly, the assistant immediately goes to the source of truth: the server argument definitions in server_args.py. By grepping for fp4_gemm_runner_backend, FP4.*backend, cutlass.*fp4, and nvfp4, the assistant is mapping out the available configuration space for FP4 kernel backends.

This is a critical design decision visible in the message: the assistant chooses to inspect the code rather than re-run with different flags blindly. By reading the source, it can understand what backends exist, what the defaults are, and how they are configured — knowledge that will inform the next fix attempt.

The grep output reveals several important pieces of information:

• Line 107 mentions "petit_nvfp4" — a different FP4 kernel variant
• Line 469 shows the default: fp4_gemm_runner_backend: str = "flashinfer_cutlass" — confirming the assistant's hypothesis that this is the active backend
• Line 607 shows disable_flashinfer_cutlass_moe_fp4_allgather: bool = False — a related but separate flag for MoE allgather operations
• Lines 4240-4241 show that fp4_gemm_runner_backend is a configurable server argument with a default value

Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are reasonable but worth examining:

Assumption 1: The NaN is caused by the FP4 GEMM runner. This is the most likely cause — FP4 kernels on a new architecture are the most fragile component. However, NaN outputs could theoretically come from other sources: the MoE routing logic, the attention mechanism, or even a corrupted model weight file. The assistant implicitly rules these out, focusing on the GEMM runner as the primary suspect. This assumption turns out to be correct, as subsequent investigation (in messages following this one) confirms that switching the MoE runner backend and FP4 GEMM backend resolves the issue.

Assumption 2: The flashinfer_cutlass backend is incompatible with SM120. This is a reasonable inference from the symptom, but it's not proven by the grep output alone. The assistant is operating on the hypothesis that CUTLASS-based kernels (NVIDIA's template library for CUDA matrix operations) may not have SM120-specific templates compiled in. This turns out to be partially correct — the issue is more nuanced, involving both the MoE runner backend and the FP4 GEMM backend needing specific selections.

Assumption 3: There might be an sgl-kernel backend to try instead. The assistant mentions "try the sgl-kernel one instead," referencing the petit_nvfp4 entry seen in the grep output. This assumption reflects a hope that the sgl-kernel package (which was built separately) might have SM120 support even if flashinfer's CUTLASS kernels don't. This turns out to be optimistic — the pre-built sgl-kernel wheel also lacks SM120-specific FP4 kernels, as discovered in subsequent messages.

Input Knowledge Required

To fully understand this message, one needs:

1. Knowledge of FP4 quantization: NVFP4 is NVIDIA's 4-bit floating-point format, which requires specialized GPU kernels for matrix multiplication. These kernels are not part of standard CUDA math libraries — they are custom implementations that must be compiled for each GPU architecture.
2. Knowledge of SGLang's architecture: SGLang separates the FP4 GEMM runner (which handles individual matrix multiplications) from the MoE runner (which handles expert routing and aggregation). Both can be configured independently.
3. Knowledge of Blackwell/SM120: The NVIDIA Blackwell architecture (compute capability 12.0) is new enough that many kernel libraries lack full support. SM120-specific bugs and incompatibilities are expected.
4. Knowledge of the deployment history: This message is part of a longer session where CUDA was upgraded to version 13 specifically to enable Blackwell features. The assistant had already patched SGLang's allreduce and symmetric memory code to support SM120.
5. Knowledge of the model architecture: Qwen3.5-397B-A17B is a hybrid MoE model with 512 experts, meaning the FP4 GEMM operations are critical for both the attention layers and the expert feed-forward networks.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

1. Confirmed default FP4 backend: The grep output definitively shows that flashinfer_cutlass is the default FP4 GEMM runner backend. This is important because it means any user deploying NVFP4 models on Blackwell will hit this same issue unless they explicitly override the backend.
2. Mapped the configuration surface: The assistant discovered the fp4_gemm_runner_backend server argument, its location in server_args.py, and the related disable_flashinfer_cutlass_moe_fp4_allgather flag. This provides a clear path for fixing the issue — override the backend via command-line flag.
3. Discovered the petit_nvfp4 option: Line 107 reveals a petit_nvfp4 string, suggesting there's an alternative FP4 kernel path. This becomes relevant later when the assistant explores building sgl-kernel from source.
4. Identified the next investigation step: The grep output doesn't show the full list of available backends (the FP4_GEMM_RUNNER_BACKEND_CHOICES list). This creates a natural next step — inspect that list to see what alternatives exist.

The Thinking Process Visible in Reasoning

The message reveals a clear chain of reasoning that is worth unpacking:

Step 1: Symptom observation. The NaN happened finish reason is recognized as a numerical computation failure, not a model loading or configuration error.

Step 2: Hypothesis formation. The assistant connects the symptom to the FP4 GEMM runner, specifically the flashinfer_cutlass backend. This hypothesis is based on domain knowledge: FP4 kernels on a new architecture (SM120) are the most likely source of numerical instability.

Step 3: Evidence gathering. Rather than acting on the hypothesis immediately, the assistant gathers evidence by inspecting the server argument definitions. This is a deliberate choice to understand the configuration space before making changes.

Step 4: Pattern matching. The assistant recognizes that this is likely the same class of issue seen with other Blackwell-incompatible defaults — similar to how the attention backend had to be changed from flashinfer to triton earlier in the deployment.

The reasoning is notably systematic. The assistant doesn't just try random flags — it reads the source code to understand what options exist, what the defaults are, and how they connect. This reflects a deep understanding that on a bleeding-edge platform like Blackwell with NVFP4, the defaults are likely wrong and must be explicitly overridden.

Mistakes and Incorrect Assumptions

While the message itself is correct in its analysis, there are some limitations worth noting:

The assumption that the fix is simple. The assistant initially hopes that simply changing the backend flag will resolve the issue. In reality, the fix requires both changing the MoE runner backend to flashinfer_cutlass and the FP4 GEMM backend to flashinfer_cudnn — and even then, the pre-built sgl-kernel may lack SM120 support entirely, requiring a full from-source rebuild. The message doesn't yet account for this complexity.

The assumption that sgl-kernel is a viable alternative. The assistant mentions trying "the sgl-kernel one instead," but the grep output doesn't actually show an sgl-kernel FP4 backend option — it shows petit_nvfp4, which is a different kernel variant within sgl-kernel. The distinction matters because the fix path involves multiple backends (MoE runner + FP4 GEMM runner) that must be set together.

Missing the MoE runner backend. At this point, the assistant is focused entirely on the FP4 GEMM runner. The MoE runner backend (which handles the expert computation) is not yet in scope. The catid gist referenced in subsequent messages reveals that both backends must be explicitly set: --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn. The NaN outputs could be caused by either backend independently.

Broader Significance

This message represents a classic debugging pattern in machine learning infrastructure: a model loads successfully, reports healthy status, but produces garbage output. The gap between "server is running" and "inference is correct" is where the most subtle bugs live — numerical issues, kernel incompatibilities, quantization artifacts.

The message also illustrates the challenges of deploying on cutting-edge hardware. Blackwell GPUs (SM120) were only recently released, and the software ecosystem — SGLang, flashinfer, sgl-kernel — is still catching up. Every component must be explicitly configured for SM120 compatibility, and the defaults (which work perfectly on Hopper/A100 GPUs) are often wrong.

The assistant's methodical approach — reading source code, understanding defaults, forming hypotheses based on architecture knowledge — is a template for debugging similar issues. Rather than treating the NaN as a black-box failure, the assistant traces it to the most likely component (FP4 GEMM) and maps the configuration space before making changes. This is the difference between guessing and diagnosing.

In the subsequent messages, this diagnosis proves correct: the assistant updates the service file with --moe-runner-backend flashinfer_cutlass --fp4-gemm-runner-backend flashinfer_cudnn, rebuilds sgl-kernel from source with SM120 support, and the model begins producing coherent text. The NaN happened error is resolved, and the Qwen3.5-397B-A17B-NVFP4 model enters production — a direct result of the diagnostic work begun in this message.