Diagnosing NaN Outputs: The FP4 GEMM Backend Investigation on Blackwell GPUs

In the ongoing deployment of the Qwen3.5-397B-A17B-NVFP4 model on a system with eight NVIDIA RTX PRO 6000 Blackwell GPUs, a critical numerical stability issue emerged. After successfully loading the 223 GB model and bringing the SGLang server to a healthy state, the assistant's first test prompts returned garbage output — repeated exclamation marks captured as reasoning content — followed by a stark "NaN happened" finish reason ([msg 5843]). This article examines a single diagnostic message ([msg 5847]) in which the assistant queried the available FP4 GEMM runner backends in SGLang's source code, a crucial step in tracing the root cause of the numerical instability.

The Message

The message is a straightforward bash command executed over SSH on the remote server:

ssh root@10.1.230.174 'grep -A5 "FP4_GEMM_RUNNER_BACKEND_CHOICES = \[" /root/sglang-main/python/sglang/srt/server_args.py'

The result returned:

FP4_GEMM_RUNNER_BACKEND_CHOICES = [
    "auto",
    "flashinfer_cudnn",
    "flashinfer_cutlass",
    "flashinfer_trtllm",
]

On its surface, this is a simple grep operation: the assistant searches a Python configuration file for the definition of a list constant that enumerates the supported FP4 (floating-point 4-bit) general matrix multiply (GEMM) runner backends. But the context surrounding this query reveals a far more intricate debugging narrative.

The Reasoning and Motivation

To understand why this message was written, one must trace the chain of failures that preceded it. The assistant had just completed building the latest SGLang main branch from source, applying SM120 (Blackwell architecture) compatibility patches, and configuring a systemd service for the Qwen3.5 model. The initial service configuration used --attention-backend flashinfer, which caused an immediate crash with the error: "triton or trtllm_mha or fa4 backend are the only supported backends on Blackwell GPUs for hybrid GDN models" ([msg 5837]). This was the first hint that Blackwell GPUs have specific backend compatibility requirements that differ from earlier architectures.

After switching to --attention-backend triton, the server started successfully and loaded the model in approximately two minutes. However, the first inference test produced gibberish: a string of exclamation marks parsed as reasoning content, with no actual text output ([msg 5842]). A second test with different parameters confirmed the worst — the model returned finish_reason: "NaN happened" ([msg 5843]). NaN (Not a Number) in floating-point computations typically indicates numerical overflow, underflow, or a fundamental incompatibility between the kernel implementation and the hardware.

The assistant's immediate hypothesis, articulated in [msg 5844], was that "the flashinfer_cutlass FP4 GEMM runner not working properly on SM120". This is a sophisticated diagnostic leap. The assistant understood that:

  1. The model uses modelopt_fp4 quantization, meaning its weights and activations are stored in 4-bit floating-point format
  2. FP4 GEMM operations require specialized kernel implementations that are hardware-specific
  3. The flashinfer_cutlass backend — which uses NVIDIA's CUTLASS library for matrix multiplication — may not have been compiled or tested for SM120 (compute capability 12.0)
  4. Numerical instability in the GEMM kernels would propagate through the entire model, producing NaN outputs The grep command in [msg 5847] was therefore a reconnaissance mission: the assistant needed to inventory the available FP4 GEMM backends before deciding which alternative to try.

Input Knowledge Required

Understanding this message requires significant domain knowledge spanning several areas. First, one must understand the concept of quantization in large language models — specifically NVFP4 (NVIDIA Floating Point 4-bit), a format that packs two 4-bit floats into a single byte, enabling massive memory savings at the cost of precision. The Qwen3.5 model at 397 billion parameters with 512 experts would be impossible to fit on eight GPUs without aggressive quantization; the NVFP4 format brings it to 223 GB, approximately 28 GB per GPU.

Second, one needs familiarity with the GEMM (General Matrix Multiply) operation, which is the computational backbone of neural network inference. In transformer models, GEMM operations dominate the feed-forward layers, attention projections, and expert routing. When a model uses FP4 quantization, every GEMM operation must be executed by a specialized kernel that can interpret the 4-bit format and produce correct results. The choice of GEMM runner backend determines which kernel library handles these operations.

Third, the message presupposes knowledge of the Blackwell GPU architecture (SM120). Blackwell introduces new tensor core capabilities and memory subsystem changes compared to Hopper (SM90) and earlier architectures. Kernel libraries like CUTLASS, cuDNN, and TRT-LLM must be specifically compiled with SM120 support to function correctly. The assistant had already discovered this the hard way with the attention backend — flashinfer's attention kernels lacked SM120 support, forcing a switch to triton.

Fourth, one must understand SGLang's server architecture, where the server_args.py file defines all command-line configuration options, their defaults, and their valid choices. The FP4_GEMM_RUNNER_BACKEND_CHOICES list is the authoritative source of which backends are compiled into the current build.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, while not explicitly shown in a separate reasoning block, is clearly visible through the sequence of actions. The pattern is one of systematic elimination: each failure narrows the search space.

The chain begins with the model loading successfully but producing nonsense. The assistant first tests whether the issue is with the reasoning parser by making a request without it — the NaN finish reason confirms it's a numerical problem, not a parsing artifact. The assistant then examines the server's default configuration, noting that fp4_gemm_runner_backend='flashinfer_cutlass' was auto-selected ([msg 5831]). The hypothesis forms: CUTLASS FP4 kernels may not support SM120.

The grep in [msg 5847] is the information-gathering step. Before trying alternatives, the assistant needs to know what's available. The result reveals four choices: auto, flashinfer_cudnn, flashinfer_cutlass, and flashinfer_trtllm. The auto setting would select a backend automatically, but it already chose flashinfer_cutlass, so auto is unlikely to help. The meaningful alternatives are flashinfer_cudnn (using NVIDIA's cuDNN library) and flashinfer_trtllm (using NVIDIA's TensorRT-LLM).

The assistant's next move, visible in the subsequent messages, is to try flashinfer_cudnn as the FP4 GEMM backend. This is a logical choice: cuDNN is NVIDIA's deep neural network library with extensive Blackwell support, and it's more likely to have been updated for SM120 than CUTLASS at this early stage of the architecture's lifecycle.

Assumptions Made

The message rests on several assumptions, most of which are reasonable but worth examining. The assistant assumes that the FP4 GEMM backend is the root cause of the NaN outputs. While this is the most likely culprit given the symptoms, other possibilities exist: the model weights could have been corrupted during download, the quantization configuration could be malformed, or there could be a bug in the model's forward pass implementation for the Qwen3.5 architecture. The assistant implicitly assumes that the model checkpoint is sound and that the issue lies in the inference software stack.

Another assumption is that the FP4_GEMM_RUNNER_BACKEND_CHOICES list in server_args.py accurately reflects which backends are actually compiled and functional. In practice, a backend could be listed as a choice but fail at runtime if its dependencies are missing or if it wasn't compiled for the target architecture. The assistant is treating this list as a menu of viable options, but each option will need to be empirically tested.

The assistant also assumes that the NaN issue is deterministic and reproducible — that changing the FP4 GEMM backend will produce consistent, non-random results. This is generally true for GPU kernels, but numerical instability can sometimes be intermittent depending on memory layout and thread scheduling.

Output Knowledge Created

This message produces concrete, actionable knowledge: the set of FP4 GEMM runner backends available in the current SGLang build. Before this query, the assistant (and the reader) only knew that flashinfer_cutlass was the default and was suspected of failure. After this query, three alternatives are known: flashinfer_cudnn, flashinfer_trtllm, and the auto selection mechanism.

This knowledge directly informs the next debugging step. The assistant will modify the systemd service to add --fp4-gemm-runner-backend flashinfer_cudnn and test whether this resolves the NaN issue. If cuDNN also fails, the fallback is flashinfer_trtllm. The auto option is unlikely to help since it already selected flashinfer_cutlass.

Beyond the immediate debugging context, this message contributes to the broader understanding of Blackwell GPU compatibility in the SGLang ecosystem. The assistant is effectively mapping out which kernel libraries support SM120 and which do not, building a compatibility matrix through empirical testing. This knowledge will be valuable for future deployments on Blackwell hardware.

Broader Context and Significance

This message sits at a pivotal moment in the deployment workflow. The assistant has successfully navigated multiple challenges — building SGLang from source, patching SM120 support into various components, configuring the model correctly, and getting the server to load. Now it faces the final hurdle: making the model actually produce correct output. The FP4 GEMM backend investigation is the critical path to a working deployment.

The choice of FP4 GEMM backend has performance implications beyond mere correctness. Different backends use different kernel implementations with varying levels of optimization for Blackwell's tensor cores. The flashinfer_trtllm backend, for example, leverages NVIDIA's TensorRT-LLM which includes Blackwell-specific optimizations like FP4 tensor core acceleration. If it works, it may be significantly faster than the cuDNN alternative. The assistant's decision to try flashinfer_cudnn first is pragmatic — cuDNN is a mature, well-tested library — but the ultimate goal is both correctness and performance.

In the broader narrative of the session, this message represents the transition from infrastructure setup (building, patching, configuring) to quality assurance (testing, debugging, tuning). The assistant is methodically working through a debugging checklist, and this grep command is a deliberate, well-motivated step in that process. It demonstrates the importance of understanding the software stack's configuration options when diagnosing numerical issues in machine learning inference — a skill that becomes increasingly critical as models adopt exotic quantization formats and target novel hardware architectures.