The Moment of Truth: Testing Qwen3.5-397B-A17B-NVFP4 on Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, few moments are as tense as the first inference call after a complex setup. Message <msg id=5842> captures exactly this moment: the assistant has just finished building the latest SGLang main branch from source, applied SM120 (Blackwell) compatibility patches, downloaded the 223 GB Qwen3.5-397B-A17B-NVFP4 model, configured a systemd service, and waited 135 seconds for the server to become healthy. Now, with a single curl command, the assistant attempts to validate the entire deployment — and what comes back is either a triumph or a disaster.

This message is a pivotal diagnostic point in the conversation. It represents the boundary between "the server is running" and "the server is working correctly." The assistant's decision to test with a simple prime-number function request, the choice of parameters (max_tokens: 200, temperature: 0), and the interpretation of the response all reveal deep knowledge of how production LLM serving works — and the assumptions that can go wrong when deploying cutting-edge hardware and quantization schemes.

Context and Motivation

To understand why this message was written, we must trace the events leading up to it. The session had just completed a major transition from serving Kimi-K2.5 INT4 (a 547 GB model with EAGLE-3 speculative decoding) to deploying a newer, more efficient model: nvidia/Qwen3.5-397B-A17B-NVFP4. This model is quantized using NVIDIA's NVFP4 format — a 4-bit floating-point quantization scheme that is only supported on Blackwell (SM120) GPUs. The assistant had:

Built the latest SGLang main branch from source (necessary because the new model architecture and modelopt_fp4 quantization path were not yet in any released version).
Applied SM120 patches to all_reduce_utils.py and torch_symm_mem.py to enable torch symmetric memory and FlashInfer allreduce fusion on compute capability 12 GPUs.
Downloaded all 19 safetensor shards totaling 223 GB.
Created a systemd service (sglang-qwen.service) with carefully chosen parameters: tp=8 (tensor parallelism across all 8 GPUs), --attention-backend triton (required for hybrid GDN models with linear attention layers on Blackwell), --quantization modelopt_fp4, and appropriate parsers (--reasoning-parser qwen3, --tool-call-parser qwen3_coder).
Waited through the 135-second loading process, which the assistant notes is "much faster than Kimi-K2.5 (223 GB vs 547 GB)." The motivation for the message is straightforward: after any model deployment, the first inference call serves as the definitive smoke test. The server reports healthy via its /health endpoint, but that only confirms the process is alive and the model weights are loaded. It does not confirm that the model produces coherent text. The assistant needs to validate end-to-end functionality before declaring the deployment successful.

The Test Design

The assistant chose a simple, well-defined prompt: "Write a Python function to check if a number is prime. Be concise." This is an excellent choice for a smoke test for several reasons:

Deterministic knowledge: Prime number checking is a fundamental programming task that any competent LLM should handle correctly. There is no ambiguity about the expected output.
Low temperature: Setting temperature: 0 forces greedy decoding, eliminating randomness as a variable. If the model produces garbage at temperature 0, the issue is systematic, not stochastic.
Short expected output: A concise prime-checking function is typically 5-15 lines of Python. With max_tokens: 200, the model has ample room but the response is quick to evaluate.
Easy to spot correctness: Either the output is valid Python that implements primality testing, or it is not. There is no gray area. The assistant also chose to pipe the output through python3 -m json.tool, which formats the JSON response prettily. This is a small but telling detail: the assistant expects to read the full response structure, including reasoning_content and tool_calls fields, to verify that all parts of the serving pipeline are working correctly.

What the Response Revealed

The response was alarming:

{
    "id": "dcbb89e0c326477687c4b880ea630d62",
    "object": "chat.completion",
    "created": 1772893606,
    "model": "qwen3.5-397b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": null,
                "reasoning_content": "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!...

The model produced content: null and reasoning_content filled with repeated exclamation marks. This is textbook garbage output — the model is generating tokens, but they are semantically meaningless. The reasoning_content field (which Qwen3.5 uses to output chain-of-thought reasoning before the final answer) is being filled with a single repeated character, and the actual response content is absent.

This pattern — repeated punctuation characters or NaN values — is a hallmark of numerical instability in quantized model inference. When FP4 (4-bit floating-point) kernels produce incorrect results due to incompatible hardware paths, the dequantization step can yield extreme values that saturate the softmax or produce garbage token IDs. The fact that the model generates something (rather than crashing outright) suggests the weights loaded correctly and the forward pass completes, but the numerical precision is catastrophically wrong at some critical point.

Assumptions and Their Consequences

The assistant made several assumptions in this message, most of which were reasonable but ultimately incorrect:

Assumption 1: The server being healthy means the model works

The assistant waited for the /health endpoint to return HTTP 200 before testing. This is standard practice, but the health check only verifies that the server process is running and the model weights are loaded into GPU memory. It does not run a forward pass or validate output quality. The assistant implicitly assumed that a successful load implies correct inference, which turned out to be false.

Assumption 2: Default FP4 GEMM and MoE backends work on SM120

The systemd service was configured with --attention-backend triton (which was explicitly chosen after an earlier crash with flashinfer), but the assistant did not explicitly set --moe-runner-backend or --fp4-gemm-runner-backend. These defaulted to auto, which on SM120 selected incompatible kernel implementations. The assistant had not yet consulted the critical reference — catid's gist documenting the exact backend flags needed for Qwen3.5 NVFP4 on Blackwell.

Assumption 3: The pre-built sgl-kernel wheel supports SM120 FP4

The assistant had sgl-kernel 0.3.21+cu130 installed as a pre-built wheel. While this wheel was compiled for CUDA 13, it may not have included SM120-specific FP4 kernels. The gist later revealed that building sgl-kernel from source with TORCH_CUDA_ARCH_LIST=12.0a and specific cmake flags is necessary for correct FP4 inference on Blackwell.

Assumption 4: Temperature 0 with a simple prompt would produce clean output

This is normally a safe assumption, but it actually helped reveal the problem more clearly. Had the assistant used a higher temperature, the garbage output might have been mistaken for creative but flawed generation. The deterministic decoding made the systematic nature of the error immediately obvious.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

LLM serving architecture: How SGLang loads models across multiple GPUs with tensor parallelism, the role of the /health endpoint, and the structure of the OpenAI-compatible chat completions API.
Quantization formats: NVFP4 is NVIDIA's 4-bit floating-point quantization format, distinct from INT4 or FP8. It requires specific kernel support in the inference engine and is only available on Blackwell (SM120) GPUs.
Blackwell GPU specifics: The RTX PRO 6000 Blackwell GPUs have compute capability 12.0 (SM120), which requires explicit support in CUDA kernels. Many libraries default to SM90 (Hopper) or lower paths.
Hybrid GDN architecture: The Qwen3.5-397B model uses a hybrid architecture with both full attention layers and linear attention (mamba-style) layers. This requires specific attention backend support — flashinfer is incompatible, and triton must be used.
MoE runner backends: SGLang supports multiple MoE kernel implementations (flashinfer_cutlass, flashinfer_cudnn, flashinfer_trtllm, etc.), and the wrong choice can produce NaN outputs on specific hardware.
The conversation's history: The assistant had just finished a multi-hour effort deploying Kimi-K2.5 with EAGLE-3 speculative decoding, and this Qwen3.5 deployment was a new pivot. The model was downloaded, the server was built from source, and this was the first validation attempt.

Output Knowledge Created

This message produced several critical pieces of knowledge:

The server is operational but the model is broken: The health check passes, the server responds to API calls, but the output is garbage. This narrows the problem space to numerical/quantization issues rather than infrastructure or loading problems.
The reasoning parser is capturing output: The reasoning_content field is populated, which means the Qwen3.5 reasoning parser is correctly intercepting the model's thinking tokens. However, the content is meaningless, suggesting the parser is working but the model's internal representations are corrupted.
The model generates tokens deterministically: With temperature 0, the model consistently produces ! characters. This rules out random sampling issues and points to a systematic numerical error in the FP4 dequantization or MoE routing.
The problem is reproducible: The assistant can re-run the test with different prompts and parameters to gather more diagnostic information (which it does in the following messages, discovering the NaN happened finish reason).

The Thinking Process

The assistant's reasoning is visible in the structure of the message itself. The opening statement — "Up in ~2 minutes — much faster than Kimi-K2.5 (223 GB vs 547 GB)" — reveals the assistant's mental model: it is comparing this deployment to the previous one, using loading time as a proxy for model size and complexity. The implicit expectation is that a smaller model should be easier to deploy.

The choice of test prompt is also revealing. The assistant could have tested with a simple "Hello" or "What is 1+1?" but instead chose a programming task. This suggests the assistant is thinking ahead to production use cases — this model will be used for coding and reasoning tasks, so testing with a code generation prompt validates the most important capability.

The assistant also chose to display the full JSON response rather than just the generated text. This indicates a systematic debugging mindset: the assistant wants to see the complete API response structure, including any error fields, finish reasons, or metadata that might reveal what went wrong.

Conclusion

Message <msg id=5842> is a classic "it compiles, but does it run?" moment in production ML engineering. The assistant successfully navigated a complex build and deployment pipeline — compiling SGLang from source, patching for SM120 compatibility, downloading a 223 GB model, configuring systemd, and waiting through the load process — only to discover that the model produces gibberish. The message captures the transition from infrastructure validation to model validation, and the diagnostic reasoning that follows.

The garbage output (repeated ! characters) was ultimately traced to incompatible default FP4 GEMM and MoE backends on SM120. The fix required explicitly setting --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn, as documented in catid's gist. This message is the critical diagnostic signal that triggered that entire debugging chain — without this test, the assistant might have declared the deployment successful based on the health check alone, and the garbage output would only have been discovered in production.

In the broader narrative of the session, this message represents the moment when the assistant learned that Blackwell FP4 inference requires non-default kernel selections — knowledge that would prove essential for future deployments on this hardware.