The Smoke Test That Proved It All: Deploying Qwen3.5-397B-A17B-NVFP4 on Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, few moments are as tense as the first smoke test after a complete infrastructure overhaul. When you've upgraded your entire software stack—PyTorch, flashinfer, SGLang, and the custom kernel library—all to bleeding-edge nightly builds, and you're serving a 397-billion-parameter model on brand-new Blackwell GPUs, the question "does it actually work?" hangs in the air. Message [msg 5926] is the answer to that question, delivered with quiet confidence: "All 5 smoke tests passed!"

This message represents the culmination of an extraordinary engineering effort spanning dozens of prior messages, multiple failed backends, a custom kernel build against an unreleased CUDA toolkit, and a critical accuracy fix that prevented silently degraded output. It is the moment where theory meets practice, where all the patches, environment variables, and compiler flags either cohere into a working system or collapse into NaN-filled garbage. In this case, they cohered.

The Message

The assistant's response at [msg 5926] reads:

[assistant] All 5 smoke tests passed! The results are excellent:

>

- Test 2: "OK" — correct, no garbage - Test 3: "323" — correct math - Test 4: Thinking mode works, reasoning content shows chain-of-thought, correct answer (323) - Test 5: 71.6 tok/s at C=1 with 512 tokens generated — very close to catid's 67.76 tok/s (we have 8 GPUs vs his 4, so slightly different dynamics)

>

The model is fully working with the nightly stack and SM120 sgl-kernel build!

Beneath this summary lies a todo list showing all seven high-priority tasks marked completed: upgrading PyTorch to nightly, upgrading flashinfer, updating SGLang main branch, applying catid's CMakeLists.txt patches to sgl-kernel, and more. The message is simultaneously a status report, a validation milestone, and a declaration of success.

The Context: A Stack-Wide Nightly Upgrade

To understand why this message matters, one must appreciate the journey that led to it. The assistant had been tasked with deploying the Qwen3.5-397B-A17B-NVFP4 model—a massive mixture-of-experts architecture with FP4 quantization—on an 8× NVIDIA RTX PRO 6000 Blackwell (SM120) server. The previous configuration used stable releases: PyTorch 2.9.1, flashinfer 0.6.4, and a pre-built sgl-kernel. The user's directive was simple but radical: "update all to nightly."

This triggered a cascade of upgrades. PyTorch went to 2.12.0.dev20260307+cu130, flashinfer to 0.6.5, and SGLang to the latest main branch. But the critical bottleneck was sgl-kernel—the custom CUDA kernel library that provides FP4 matrix multiplication and MoE routing for Blackwell GPUs. The pre-built wheel lacked SM120 (Blackwell) support, so the assistant had to build it from source, applying patches from catid (a community developer) to fix CMake policy guards, CUDA 13 cccl include paths, and FlashAttention-3 fallback handling. The build itself required careful tuning: TORCH_CUDA_ARCH_LIST=12.0a to enable FP4 kernels, MAX_JOBS=20 to avoid memory exhaustion, and -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to work around dlpack's outdated CMake requirements.

The Backend Gauntlet

Before the smoke test could even be attempted, the assistant had to find a working backend configuration. This was not straightforward. On SM120, several backends proved incompatible:

The Accuracy Crisis: FP8 KV Cache

Perhaps the most important decision embedded in this success story is invisible from the smoke test results alone. During the configuration phase, the assistant identified a critical accuracy issue: the checkpoint's default FP8 KV cache was being applied without proper scaling factors. On Blackwell GPUs, the FP8 KV cache path was untested and could silently degrade output quality, particularly for long-context agentic coding tasks where precision matters most.

The assistant's response was to explicitly force the KV cache to BF16 via --kv-cache-dtype bf16. This provided approximately 1.57 million tokens of high-precision cache—far more than needed for the intended workload—at the cost of some additional memory usage. This decision reflects a core philosophy: correctness over performance. The FP8 path might have been faster, but if it produced subtly wrong answers, the entire deployment would be unreliable. The smoke test results—"correct, no garbage"—validated this choice.

What Each Test Proves

The five smoke tests were carefully designed to probe different aspects of the deployment:

Test 1 (Model Info) verified that the server loaded the correct model path and could respond to basic API queries. This is the simplest test, but failure here would indicate a fundamental loading or routing issue.

Test 2 (Simple completion) asked for a trivial response ("OK") and got it in 0.11 seconds. This confirmed that the basic inference pipeline—tokenization, forward pass through all layers, MoE routing, FP4 dequantization, and sampling—produced coherent output. "No garbage" is a deliberately low bar, but one that many early Blackwell deployments failed to clear.

Test 3 (Math, non-thinking) asked for the product of 17 and 19 and received "323" in 0.13 seconds. This is a factual correctness test. The model could have produced "323" by memorization, but the speed and consistency suggested genuine computation. More importantly, it confirmed that the FP4 quantization did not degrade numerical reasoning.

Test 4 (Thinking mode) exercised the model's chain-of-thought capability. The assistant had to fix the test script after an initial failure caused by the API returning reasoning content in a separate field from the response content. The corrected test showed a full reasoning trace: "Thinking Process:\n\n1. Analyze the Request:\n * Task: Calculate the product of 17 and 19..." This confirmed that the thinking/reasoning parser was correctly integrated and that the model could access its full capabilities.

Test 5 (Throughput) measured 71.6 tok/s at concurrency 1 with 512 tokens generated. This is the performance validation. The assistant compared this to catid's published result of 67.76 tok/s on a 4-GPU setup, noting that the 8-GPU configuration would have slightly different dynamics (more GPUs means more communication overhead but also more memory bandwidth). The close agreement validated that the custom kernel build was not imposing a performance penalty.

The Thinking Process

The message reveals a sophisticated reasoning process in its brevity. The assistant is not merely reporting results; it is interpreting them against a known baseline (catid's numbers), acknowledging the differences in hardware configuration, and drawing a conclusion about the system's readiness. The phrase "very close to catid's 67.76 tok/s (we have 8 GPUs vs his 4, so slightly different dynamics)" shows an understanding of distributed inference scaling: more GPUs introduce NCCL all-reduce overhead that can offset raw compute gains, especially at low concurrency.

The todo list embedded in the message serves a dual purpose. For the assistant, it is a working memory of completed tasks—a way to track progress across a long session. For the reader (the user), it provides transparency into what was accomplished. The fact that all items are marked "completed" signals that the nightly upgrade project is finished.

Assumptions and Decisions

Several assumptions underpin this message:

  1. The smoke tests are representative. The assistant assumes that passing five specific tests implies general correctness. This is reasonable for a smoke test but not exhaustive—edge cases like very long contexts, tool calling, or multi-turn conversations remain untested.
  2. BF16 KV cache is sufficient. The assistant assumes that 1.57M tokens of BF16 cache meets the workload requirements. For agentic coding tasks with long conversations, this is likely true, but it is an assumption worth noting.
  3. catid's results are a valid baseline. The assistant treats catid's published throughput as a reference point, implicitly trusting the methodology and hardware configuration. The comparison is informative but not rigorous—different GPU counts, model versions, and prompt lengths could skew results.
  4. The nightly stack is stable enough for production. By deploying via systemd with the nightly build, the assistant implicitly endorses the stability of PyTorch 2.12.0.dev, flashinfer 0.6.5, and the custom sgl-kernel. Nightly builds carry inherent risk of regressions, but the smoke tests provide some confidence.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with the SGLang serving stack, awareness of Blackwell (SM120) GPU architecture challenges, understanding of FP4 quantization and its implications for model quality, knowledge of NCCL all-reduce overhead in multi-GPU inference, and familiarity with catid's community benchmarks for the same model family.

Output knowledge created by this message is significant. It confirms that the Qwen3.5-397B-A17B-NVFP4 model can be successfully deployed on Blackwell GPUs using a fully nightly software stack. It validates the specific backend configuration (flashinfer_cutlass for MoE, flashinfer_cudnn for FP4 GEMM) as a working combination. It establishes a performance baseline (~72 tok/s at C=1) for future optimization work. And it demonstrates that the custom sgl-kernel build with SM120 FP4 support produces correct numerical output.

Conclusion

Message [msg 5926] is a moment of arrival. After the uncertainty of kernel compilation, the frustration of crashing backends, and the vigilance required to catch the FP8 KV cache pitfall, the assistant can finally say: "The model is fully working." The five smoke tests are not just checkmarks on a list—they are evidence that every layer of the stack, from CUDA 13's compiler to the model's attention heads, is functioning correctly on this unprecedented hardware.

The message's understated tone belies the engineering drama behind it. There is no triumphalism, no extended reflection—just a clear summary of results and a quiet acknowledgment that the system now meets its requirements. For anyone who has ever deployed a large language model on novel hardware, this message resonates: the relief when the first correct token emerges, the validation when the throughput numbers align with expectations, and the quiet satisfaction of marking the last todo item as complete.