The Smoke Test That Validated Everything: Correctness Verification After a Stack-Wide Nightly Upgrade

Introduction

In the midst of a high-stakes deployment of the Qwen3.5-397B-A17B-NVFP4 model on an 8× NVIDIA RTX PRO 6000 Blackwell (SM120) server, message [msg 5923] represents a pivotal moment of validation. After an exhaustive multi-hour effort to upgrade the entire software stack—PyTorch to nightly 2.12.0+cu130, flashinfer to 0.6.5, SGLang to the latest main branch, and a custom-built sgl-kernel from source with SM120 FP4 support—the assistant executes a smoke test script to verify that the model actually produces correct output. This single message captures the tension between the complexity of the infrastructure work that preceded it and the fundamental question that all such work ultimately answers: does it actually work?

The message is deceptively simple on its surface: a bash command that copies a Python script to the remote server and executes it. But the output it returns—three passing tests and one failing test—carries enormous weight. It represents the first confirmation that the entire chain of nightly builds, patched source code, and hand-tuned compilation flags has produced a working inference server capable of serving a 397-billion-parameter model on cutting-edge Blackwell hardware.

The Context: A Stack-Wide Nightly Upgrade

To understand why this message was written, one must appreciate the journey that led to it. The assistant had been tasked with deploying Qwen3.5-397B-A17B-NVFP4—a massive mixture-of-experts model using NVIDIA's FP4 (4-bit floating point) quantization format—on a server equipped with eight RTX PRO 6000 Blackwell GPUs. These GPUs use the SM120 compute architecture, which is so new that much of the software ecosystem does not yet support it natively.

The user's directive was simple: "update all to nightly." This kicked off a cascade of work spanning dozens of messages ([msg 5896] through [msg 5922]). The assistant upgraded PyTorch to 2.12.0.dev20260307+cu130 (a nightly build from March 7, 2026), upgraded flashinfer to 0.6.5, pulled the latest SGLang main branch, and—most critically—built sgl-kernel from source with SM120 support. This last step required applying patches from a developer named "catid" that added CMake policy guards for CUDA 13 compatibility, fixed cccl include paths, and provided a soft fallback for the FlashAttention-3 (FA3) module that doesn't yet support Blackwell. The build was compiled with TORCH_CUDA_ARCH_LIST=12.0a to enable FP4 kernels, and required setting CUDACXX explicitly and adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to work around dlpack's outdated CMake configuration.

After the build succeeded, the assistant installed the wheel, verified imports, updated the systemd service file with the correct backend flags (using flashinfer_cutlass for the MoE runner and flashinfer_cudnn for FP4 GEMM), started the server, and waited for it to become healthy. Only then—after all that infrastructure work—did the assistant write and execute the smoke test that appears in [msg 5923].

The Smoke Test: What It Tests and Why

The smoke test script (smoke_test_qwen.py) was designed to answer four questions, each building on the previous:

  1. Is the model loaded correctly? (Test 1: Model Info) — This simply queries the model path to confirm the server recognized the checkpoint.
  2. Can the model generate coherent text? (Test 2: Simple completion) — Sending a trivial prompt and expecting the response 'OK' verifies that the basic generation pipeline works end-to-end.
  3. Can the model perform reasoning? (Test 3: Math, non-thinking) — A math problem ("What is 300+23?") tests whether the model produces correct output, not just any output. The expected answer 323 confirms numerical reasoning capability.
  4. Can the model use its thinking/reasoning mode? (Test 4: Math, thinking mode) — This tests the more complex thinking/reasoning pathway, which is essential for the model's intended use in agentic coding tasks. The results are revealing. Tests 1–3 pass cleanly. The model responds with 'OK' in 0.44 seconds and correctly computes 323 in 0.21 seconds. These are not just performance numbers—they are correctness guarantees. After all the patching, building, and configuration, the model produces the right answers.

The Failure That Wasn't a Failure

Test 4 fails with a traceback. But crucially, this failure is not a model error—it is a test script error. The thinking mode in the Qwen3.5 API returns the reasoning content in a separate field from the response content, so accessing response.content returns None. The assistant immediately recognizes this in the following message ([msg 5924]): "Tests 1-3 passed perfectly — no garbage, correct answers! Test 4 failed because thinking mode returns content differently (the response content is None when reasoning is in a separate field)."

This distinction matters. The assistant had previously encountered NaN and garbage outputs when testing incompatible backend configurations (the flashinfer_trtllm and flashinfer_cutedsl backends crashed or produced garbage on SM120). The fact that Tests 1–3 produce clean, correct answers is the real signal. The Test 4 failure is a mundane API compatibility issue in the test script, not a model inference bug.

Assumptions and Knowledge

The smoke test makes several implicit assumptions. First, it assumes that the server's health endpoint accurately reflects readiness—the assistant waited for /health to return 200 before running the test. Second, it assumes that simple completions are representative of overall model quality; while not a rigorous evaluation, a model that produces garbage on simple prompts would certainly fail here. Third, it assumes the test script correctly handles the API response format, which turns out to be incorrect for thinking mode—a reasonable assumption given that the API format had changed between SGLang versions.

The input knowledge required to understand this message includes familiarity with the SGLang inference server architecture, the Qwen3.5 model family's API conventions (particularly the separation of reasoning content), and the broader context of the Blackwell SM120 compatibility challenges. The output knowledge created is the critical validation that the nightly stack upgrade produced a working inference server—a result that justifies the hours of build debugging and patching that preceded it.

The Thinking Process Visible in the Message

The structure of the smoke test itself reveals the assistant's thinking process. The tests are ordered from simplest to most complex: first verify the model loads, then verify basic generation, then verify reasoning, then verify thinking mode. This is a textbook debugging approach—eliminate the simplest failure modes first before testing more complex pathways.

The choice of test prompts is also deliberate. "Say 'OK'" is a minimal test that requires almost no reasoning capacity—it tests the generation pipeline. "What is 300+23?" tests actual mathematical reasoning but is simple enough that any failure would clearly indicate a model problem rather than a prompt misunderstanding. The thinking mode test extends this to the more complex reasoning pathway that the model will use in production for agentic coding tasks.

Conclusion

Message [msg 5923] is the payoff moment in a long chain of infrastructure work. It transforms the abstract question of "did the build succeed?" into the concrete answer of "the model produces correct output." The three passing tests provide confidence that the nightly stack upgrade was successful, while the single failing test (correctly diagnosed as a test script issue) demonstrates the assistant's ability to distinguish between real model problems and superficial API quirks. This message marks the transition from build-and-configure mode to validation-and-production mode—a critical inflection point in any deployment workflow.