The Smoke Test That Caught a Silent Killer: When Speed Masks Garbage on Blackwell GPUs

In the high-stakes world of deploying large language models on novel hardware architectures, few moments are as revealing as the first inference call. On March 7, 2026, an AI assistant working on an 8× RTX PRO 6000 Blackwell (SM120) GPU cluster experienced exactly such a moment. After spending hours upgrading the CUDA stack, patching SGLang for SM120 support, building custom kernels from source, and methodically testing backend after backend, the assistant issued a smoke test — and the result was devastating. The model responded with sixteen exclamation marks: !!!!!!!!!!!!!!!!. The smoke test's garbage detector caught it instantly, and the assistant was forced to confront a hard truth: the fastest backend yet discovered was producing nothing but noise.

The Message: A Single Bash Command That Changed Everything

The subject message ([msg 5966]) is deceptively brief. The assistant writes simply:

Fast! Let me run the smoke test: [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 /root/smoke_test_qwen.py' === Test 1: Model Info === model: /data/models/Qwen3.5-397B-A17B-NVFP4

>

=== Test 2: Simple completion (non-thinking) === response: '!!!!!!!!!!!!!!!!' time: 0.59s, usage: {'prompt_tokens': 17, 'total_tokens': 33, 'completion_tokens': 16, 'prompt_tokens_details': None, 'reasoning_tokens': 0} Traceback (most recent call last): File "/root/smoke_test_qwen.py", line 37, in <module> assert "!" * 10 not in resp, "GARBAGE DETECTED (repeated !)" ^^^^^^^^^^^^^^^^^^^^ AssertionError: GARBA...

The single word "Fast!" is laden with dramatic irony. The server started in roughly 30 seconds — the health check succeeded on the third attempt — and the first inference completed in just 0.59 seconds. But speed without correctness is worthless. The smoke test script, purpose-built for this exact scenario, detected the telltale pattern of repeated exclamation marks and raised an assertion error. The model was producing garbage.

The Long Road to This Moment

To understand why this message carries such weight, one must appreciate the journey that led to it. The assistant had been engaged in a multi-session optimization campaign spanning dozens of messages and hours of work. The machine in question — an 8-GPU node with RTX PRO 6000 Blackwell GPUs — represented a cutting-edge but poorly supported hardware target. Blackwell (compute capability SM120) was so new that most ML frameworks had not yet been updated to support it.

The assistant had already:

• Upgraded the entire software stack to nightly builds, including PyTorch 2.12.0.dev20260307+cu130 and FlashInfer 0.6.5
• Built sgl-kernel from source with SM120 patches applied, compiling with TORCH_CUDA_ARCH_LIST=12.0a to enable FP4 kernels
• Tested and eliminated flashinfer_trtllm as a MoE backend (it crashed because TRT-LLM kernels target SM100, not SM120)
• Discovered that flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM produced correct output
• Identified and fixed an FP8 KV cache accuracy issue by forcing BF16 The flashinfer_cutedsl backend represented a promising alternative. CUTE DSL (CUDA Template Extensions Domain-Specific Language) is a JIT-compilation approach that generates kernels specifically for the target architecture at runtime. Unlike pre-compiled TRT-LLM kernels that hardcode SM100 support, CUTE DSL should theoretically adapt to any CUDA-capable architecture. When the server launched successfully and began CUDA graph capture — the lengthy process of recording CUDA kernel launches for later replay — the assistant had reason to be optimistic. The log output showed GDN kernel dispatchers initializing across all 8 TP (tensor parallelism) ranks, and the health check passed quickly. "Fast!" was an expression of genuine relief.## The Reasoning Behind the Test The assistant's decision to run the smoke test immediately after the server became healthy reflects a disciplined engineering approach. Throughout the optimization campaign, the assistant had developed a standard smoke test script (/root/smoke_test_qwen.py) that served as a canary for correctness. The script tested multiple scenarios — simple completions, reasoning/thinking mode, tool calls — and included assertions to catch specific failure modes. The "repeated exclamation marks" pattern was a known artifact of FP4 quantization issues on SM120, first encountered earlier in the session when the default auto backend produced identical garbage output. The assistant's thinking process, visible in the preceding messages, reveals a systematic elimination strategy. After flashinfer_trtllm crashed with a kernel compilation error (it tried to compile for SM100 on SM120 hardware), the assistant pivoted to flashinfer_cutedsl with the reasoning: "it should JIT-compile for SM120." This was a reasonable assumption — CUTE DSL's JIT nature should make it architecture-agnostic. The server launched successfully, CUDA graph capture began, and the health check passed. All signals pointed to success. But the assumption was wrong. The smoke test exposed the truth: flashinfer_cutedsl suffered from the same SM120 kernel correctness issue as the default auto backend. The JIT compilation produced kernels that ran without crashing but computed incorrect results. This is arguably worse than a crash — a crash tells you something is broken, while silent numerical errors can go undetected for months in production.

The Input Knowledge Required

To fully understand this message, one needs substantial context about the ML inference stack:

1. FP4 quantization (NVFP4/NVIDIA FP4): A 4-bit floating point format used to compress model weights. The Qwen3.5-397B-A17B model uses NVIDIA's FP4 format, which requires specialized GPU kernels for matrix multiplication. These kernels are architecture-specific and must be compiled for the target GPU's compute capability.
2. SM120 vs SM100: Blackwell GPUs have compute capability 12.0 (SM120), while the previous Hopper generation is SM90 and the Blackwell preview/engineering samples were SM100. Many existing kernels target SM100 and fail or produce incorrect results on SM120.
3. MoE (Mixture of Experts) backends: The model uses a Mixture of Experts architecture with 17 active experts out of 397B total parameters. SGLang supports multiple MoE runner backends — flashinfer_trtllm (fused TRT-LLM kernels), flashinfer_cutlass (CUTLASS library), flashinfer_cutedsl (CUTE DSL JIT), and triton (Triton compiler). Each has different architecture support and performance characteristics.
4. CUDA graph capture: A performance optimization that records a sequence of CUDA kernel launches into a replayable graph, reducing CPU launch overhead. This process is time-consuming and can mask kernel correctness issues because the graph capture succeeds even if the kernels compute wrong results.
5. The smoke test script: A custom Python script that sends test prompts and validates responses. The garbage detector (assert "!" * 10 not in resp) catches a specific failure mode where FP4 quantization produces repeated punctuation characters — a pattern observed in earlier backend tests.

The Output Knowledge Created

This message produced several critical pieces of knowledge:

1. flashinfer_cutedsl is incompatible with SM120 for FP4 MoE: Despite successfully launching and passing CUDA graph capture, the CUTE DSL backend produces garbage output. This eliminates another candidate from the list of viable backends.
2. The garbage pattern is consistent across multiple backends: Both the default auto backend and flashinfer_cutedsl produce the same !!!!!!!! garbage pattern. This suggests a common root cause — likely the FP4 GEMM or MoE kernel dispatch logic in FlashInfer's SM120 support — rather than a bug specific to one backend.
3. Successful server launch does not guarantee correct inference: The server passed health checks, initialized all 8 TP ranks, completed CUDA graph capture, and responded to inference requests in 0.59 seconds — yet produced garbage. This is a sobering reminder that health checks only verify process liveness, not output quality.
4. flashinfer_cutlass remains the only known working backend: From earlier testing, the assistant had established that flashinfer_cutlass for MoE combined with flashinfer_cudnn for FP4 GEMM produced correct output. The cutedsl experiment was an attempt to find a faster alternative, and it failed.

The Mistake and Its Implications

The assistant made a reasonable but incorrect assumption: that CUTE DSL's JIT compilation would automatically produce correct kernels for SM120. The assumption was based on the architecture of CUTE DSL — it uses CUDA's template metaprogramming to generate kernels at runtime, which should theoretically adapt to any target architecture. However, the reality is more nuanced. The kernel templates themselves may contain hardcoded assumptions about warp sizes, shared memory layouts, or instruction selection that are valid for SM100 but incorrect for SM120. The JIT compiler can only work with the templates it has; it cannot fix fundamental algorithmic assumptions baked into the template code.

This mistake highlights a broader challenge in deploying ML models on new hardware: the gap between "it compiles and runs" and "it produces correct results." Traditional software testing often focuses on crashes and errors, but numerical correctness in GPU kernels is far more subtle. A kernel can execute without any runtime errors, consume the correct amount of memory, and complete in the expected time — yet compute entirely wrong results due to an incorrect instruction selection or data layout assumption.

The Aftermath

The assistant's immediate response, visible in the following message ([msg 5967]), was to acknowledge the failure and pivot: "Garbage output with flashinfer_cutedsl! The !!!! pattern is back. So flashinfer_cutedsl has the same SM120 kernel issue as the default auto backend. Only flashinfer_cutlass works correctly."

The assistant then cleaned the GPUs and prepared to test a new combination: flashinfer_cutlass for MoE (known working) with flashinfer_trtllm for the dense FP4 GEMM (the --fp4-gemm-backend flag, distinct from the MoE runner backend). This represents a pragmatic engineering response — when the fastest option fails, fall back to what works, but continue exploring incremental improvements.

Conclusion

Message [msg 5966] captures a pivotal moment in a complex optimization campaign. In just a few lines of output, it invalidates hours of work, eliminates a promising backend candidate, and reinforces a hard-learned lesson about the gap between "it runs" and "it works." The smoke test — a simple assertion checking for repeated exclamation marks — proved more valuable than any benchmark or health check. It caught a silent correctness failure that would have otherwise gone undetected, potentially corrupting every inference served by the model.

For anyone deploying ML models on novel hardware, this message serves as a cautionary tale: trust your smoke tests, not your launch logs. Speed is seductive, but correctness is non-negotiable.