The Diagnostic Pivot: Tracing NaN Outputs to FP4 Backend Incompatibility on Blackwell

In the middle of deploying a cutting-edge 397-billion-parameter Mixture-of-Experts model on eight Blackwell GPUs, a single bash command — barely a dozen characters of grep pattern — captured a moment of critical diagnostic reasoning. The message in question ([msg 5846]) is deceptively simple:

[assistant] [bash] ssh root@10.1.230.174 'grep "FP4_GEMM_RUNNER_BACKEND" /root/sglang-main/python/sglang/srt/server_args.py | head -5'
FP4_GEMM_RUNNER_BACKEND_CHOICES = [
    FP4_GEMM_RUNNER_BACKEND_CHOICES.extend(choices)
            choices=FP4_GEMM_RUNNER_BACKEND_CHOICES,

This tiny grep, executed against a remote server's SGLang source tree, represents the culmination of a diagnostic chain that had begun minutes earlier with a far more alarming signal: the model was producing nothing but exclamation marks and NaN errors.

The Context: A Model That Speaks Gibberish

The story begins with a successful deployment. The assistant had just finished building the latest SGLang main branch from source, applied SM120 compatibility patches for Blackwell GPUs, downloaded the 223 GB nvidia/Qwen3.5-397B-A17B-NVFP4 model, and launched it as a systemd service with tensor parallelism across all eight GPUs ([msg 5828]-[msg 5841]). The model loaded in roughly two minutes — impressively fast for a 397B-parameter model, thanks to the NVFP4 quantization that compressed it to 223 GB.

But the first inference test ([msg 5842]) revealed a catastrophe: the model's output consisted entirely of repeated exclamation marks, captured as reasoning_content. A second test with different parameters ([msg 5843]) was even more explicit — the API returned "finish_reason": "stop" with "matched_stop": "NaN happened". The model was producing numerical garbage.

This is the kind of failure that stops a deployment cold. NaN (Not a Number) values in neural network computations indicate that floating-point operations have produced undefined results — typically from division by zero, infinity propagation, or arithmetic on uninitialized values. In the context of quantized models on new GPU architectures, the most likely culprit is a kernel incompatibility: the GPU's compute capability (SM120, Blackwell) is not properly supported by the matrix multiplication kernels selected for the quantized weights.

The Diagnostic Chain

The assistant's reasoning process, visible across the preceding messages, followed a logical progression. First, it ruled out trivial causes: the model loaded without errors, the server started cleanly, and the health endpoint responded. The problem was not in loading or serving infrastructure but in the actual computation.

The assistant then hypothesized that the FP4 (4-bit floating-point) GEMM (General Matrix Multiply) runner backend — the component responsible for executing quantized matrix multiplications — was incompatible with Blackwell's SM120 architecture. The default backend, flashinfer_cutlass, had been auto-selected ([msg 5831]), but its CUTLASS-based FP4 kernels might not support SM120.

This hypothesis was grounded in prior experience. Throughout this session, the assistant had repeatedly encountered SM120 compatibility gaps — in NCCL tuning dictionaries, in FlashInfer allreduce fusion, in Torch symmetric memory, and in attention backends. Each time, the pattern was the same: SGLang's codebase had hardcoded dictionaries or conditional branches that covered compute capabilities 8, 9, and 10 (Hopper, Ada, and earlier Blackwell variants) but omitted capability 12 (the specific Blackwell implementation in these RTX PRO 6000 GPUs). The FP4 GEMM runner was simply the latest in a long line of components that needed SM120 patches.

The Subject Message: A Targeted Code Probe

The grep command in [msg 5846] was the assistant's attempt to enumerate the available FP4 GEMM runner backends in SGLang's server argument system. The reasoning was straightforward: if the default flashinfer_cutlass backend produces NaN on SM120, perhaps an alternative backend — flashinfer_cudnn or flashinfer_trtllm — would work correctly.

The choice of server_args.py as the target file was deliberate. In SGLang's architecture, server arguments are defined with their types, defaults, and valid choices in this centralized configuration module. By grepping for FP4_GEMM_RUNNER_BACKEND, the assistant expected to find the FP4_GEMM_RUNNER_BACKEND_CHOICES list that enumerates valid backend names.

However, the grep output was incomplete. The pattern matched the list declaration header (FP4_GEMM_RUNNER_BACKEND_CHOICES = [) and the function that extends it, but not the actual choices values. This is because head -5 truncated the output before the list contents, and the grep pattern was too narrow to capture multi-line list definitions. The assistant would need a follow-up command ([msg 5847]) with a broader pattern to see the full list: "auto", "flashinfer_cudnn", "flashinfer_cutlass", "flashinfer_trtllm".

Assumptions and Their Consequences

The assistant made several assumptions in this message. First, it assumed that the FP4 backend choices would be defined as a static Python list in server_args.py — a reasonable assumption given SGLang's code organization, and one that proved correct. Second, it assumed that the variable name would contain the exact string FP4_GEMM_RUNNER_BACKEND, which was also correct.

A more subtle assumption was that the problem lay in the GEMM runner backend rather than elsewhere in the computation pipeline — for example, in the MoE (Mixture-of-Experts) runner backend, which handles the expert routing and aggregation for models like Qwen3.5 with 512 experts. This assumption would later be refined when the user shared a gist ([msg 5850], [msg 5852]) revealing that both --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn needed to be set explicitly for SM120 compatibility ([msg 5854]). The assistant's initial focus on only the FP4 GEMM backend was therefore incomplete — the MoE runner also required explicit configuration.

Input and Output Knowledge

To understand this message, the reader needs knowledge of several domains: the SGLang serving framework and its server argument system; the concept of FP4 quantization and GEMM runner backends; the Blackwell GPU architecture (SM120) and its compatibility challenges; and the symptom of NaN outputs indicating numerical instability in quantized kernels.

The message produced a modest but essential output: confirmation that the FP4_GEMM_RUNNER_BACKEND_CHOICES list exists in server_args.py and that the assistant was on the right track. The actual choices values remained hidden, requiring the follow-up command in [msg 5847] to reveal them. This is a pattern common in exploratory debugging — each command narrows the search space incrementally, and partial results guide the next step.

The Broader Significance

This message, for all its brevity, exemplifies the systematic diagnostic approach that characterizes effective ML infrastructure work. The assistant did not restart the server randomly or try opaque configuration changes. Instead, it traced the NaN symptom to its most probable root cause (FP4 kernel incompatibility), identified the configuration mechanism (FP4 GEMM runner backend choices), and probed the codebase to enumerate alternatives.

The follow-up was swift. After discovering the available backends and receiving the user's gist with SM120 instructions, the assistant applied the fix: --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn ([msg 5854]). This combination — using CUTLASS for MoE operations and cuDNN for FP4 GEMM operations — resolved the NaN issue and produced correct model outputs.

In the broader arc of the session, this message sits at a transition point between deployment and debugging. The Kimi-K2.5 model had been hardened into a production service with systemd, hierarchical KV cache, and tool-call parsing. The Qwen3.5 deployment was intended to follow the same pattern, but the NaN outputs forced a detour into low-level kernel compatibility. The grep command in [msg 5846] was the first step of that detour — a small but essential probe into the machinery of quantized computation on next-generation hardware.