The Three-Word Message That Saved a Deployment

Subject message: [user] https://gist.github.com/catid/87cca824963f17fe7479a0ed26221397 - sm120 instructions

At first glance, the message is almost absurdly brief. A URL and three words: "sm120 instructions." In a conversation spanning thousands of messages across dozens of hours of work — environment setup, driver installation, CUDA upgrades, kernel compilation, model downloading, and intricate speculative decoding optimization — this single line from the user at message 5852 looks like the smallest of gestures. But it is anything but. This message is the pivotal turning point in a protracted debugging crisis, the key that unlocked a deployment that had been stalled by silent numerical corruption. To understand why, one must trace the arc of failure that preceded it.

The NaN Crisis

In the messages immediately before this one, the assistant had been attempting to deploy a new model — nvidia/Qwen3.5-397B-A17B-NVFP4 — on an 8-GPU system of NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. This was a significant model: 397 billion total parameters, 17 billion active, using NVFP4 (4-bit floating point) quantization for efficient inference. The assistant had built the latest SGLang main branch from source, applied SM120 patches for all-reduce and symmetric memory, and started the server. The model loaded. And then it produced garbage.

The first test returned a response consisting entirely of repeated exclamation marks — hundreds of them — captured as "reasoning_content." The second test returned null content with a finish reason of "NaN happened". The model was producing numerical garbage, and the root cause was opaque. The assistant tried changing the attention backend from flashinfer to triton (which fixed a separate crash about hybrid GDN models on Blackwell), but the NaN issue persisted. The server logs showed no obvious errors. The model weights had loaded correctly. The quantization was detected as modelopt_fp4. Everything looked right, and yet the output was nonsense.

This is the kind of bug that can consume days. Numerical issues in FP4 kernels on a new GPU architecture (SM120, Blackwell's compute capability 12.0) could stem from any number of places: incorrect kernel selection, missing SM120 code paths in the CUDA kernels, ABI mismatches between compiled extensions and PyTorch, or simply the wrong default backend being chosen at runtime. Without external reference, the assistant was reduced to probing the codebase for available backend options and guessing.

The User's Intervention

The user had already posted the same bare URL once before, at message 5850, without any annotation. The assistant produced an empty response — it appears the webfetch tool was not invoked, or the message was not processed. So the user reposted, this time adding the crucial annotation: "sm120 instructions." This small addition transforms the message. It tells the assistant: this is not just a random link; this is a recipe specifically for the hardware you're fighting with.

The user's motivation is clear: they possess external knowledge — a GitHub Gist authored by someone named "catid" who has already solved this exact problem on the same hardware. Rather than attempting to guide the assistant through the debugging process step by step, the user provides the definitive reference. This is a pattern that recurs throughout the conversation: the user acts as a source of high-signal external information, while the assistant handles the mechanical execution. The user knows the territory; the assistant knows the tools.

What the Gist Contained

The Gist at that URL documented the exact configuration needed to run Qwen 3.5 NVFP4 on SM120 Blackwell GPUs. Its critical revelations were:

  1. The MoE runner backend must be explicitly set to flashinfer_cutlass — the default auto selection picks a kernel that produces NaN on SM120.
  2. The FP4 GEMM runner backend must be explicitly set to flashinfer_cudnn — similarly, the default is broken on Blackwell.
  3. The attention backend must be triton for hybrid GDN models (models mixing linear attention and full attention layers), which the assistant had already discovered independently.
  4. The sgl-kernel package may need to be built from source with SM120-specific compilation flags: TORCH_CUDA_ARCH_LIST=12.0a and CMake flags enabling FP4 support while disabling pre-SM90 kernels. These three backend flags — --moe-runner-backend flashinfer_cutlass, --fp4-gemm-runner-backend flashinfer_cudnn, and --attention-backend triton — were the missing pieces. The assistant had been running with default backend selection, which on SM120 silently selected incompatible kernel implementations.

Assumptions and Missteps

Several assumptions had led the assistant into this dead end. The first was that the default backend selection in SGLang would choose correct implementations for SM120. This assumption was reasonable — SGLang is a mature project with broad GPU support — but it failed because Blackwell (compute capability 12.0) is still new, and not all code paths have been validated. The auto backend selector likely fell through to a fallback that happened to work on older architectures but produced silent numerical corruption on SM120.

The second assumption was that the pre-built sgl-kernel wheel (version 0.3.21+cu130) would contain SM120-compatible FP4 kernels. In reality, pre-built wheels may target the most common architectures (SM80, SM90) and omit SM120-specific kernels to keep build times manageable. The assistant had not considered this possibility.

The third, more subtle assumption was that a model loading without errors and producing any output meant the kernels were functionally correct. The "NaN happened" finish reason was a clear signal, but the assistant initially treated it as a transient issue rather than a fundamental backend incompatibility.

The user's message implicitly corrects all three assumptions. By providing a Gist that documents exact backend flags and a from-source build process, the user signals: the defaults are wrong for this hardware, and you need to be explicit.

Input Knowledge Required

To understand this message, the reader needs to know several things. First, that "SM120" refers to the compute capability of NVIDIA Blackwell GPUs — a new architecture that requires specific software support. Second, that "NVFP4" is a 4-bit floating point quantization format that requires specialized GPU kernels for efficient inference. Third, that "MoE" (Mixture of Experts) models like Qwen3.5-397B-A17B use a router to select which expert sub-networks to activate per token, and the kernel that performs this expert computation (the "MoE runner backend") can produce incorrect results if the wrong implementation is selected. Fourth, that SGLang is a serving framework for large language models, and its backend selection logic may not yet have complete SM120 support. Finally, that a GitHub Gist is a lightweight, single-file document — the user is pointing to a concise, authoritative recipe, not a lengthy tutorial.

Output Knowledge Created

The message's immediate effect was to trigger a webfetch call by the assistant (message 5853), which retrieved the Gist content. The assistant then extracted the critical backend flags and immediately acted on them: stopping the server, updating the systemd service configuration, and restarting with --moe-runner-backend flashinfer_cutlass --fp4-gemm-runner-backend flashinfer_cudnn --attention-backend triton. This resolved the NaN issue and produced correct model output.

The message also created lasting knowledge: the assistant incorporated the Gist's findings into its running documentation (message 5856), which served as the session's master reference document. The backend flags became part of the production configuration. And the user's subsequent message (5857) — "Update all to nightly, the patches are more recent than our build" — built directly on this new understanding, triggering a full stack upgrade to nightly PyTorch and a from-source sgl-kernel build.

The Thinking Process

The user's thinking in this message is economical and precise. They had already posted the URL once without annotation and received no useful response. Rather than escalating with frustration or verbose explanation, they simply added context: "sm120 instructions." This tells the assistant what kind of thing the link is — a procedural guide, not background reading — and why it matters — it addresses the exact hardware problem stalling the deployment.

The user is also making a judgment about the assistant's capabilities. They assume the assistant can fetch and parse a web page, extract relevant technical instructions, and apply them to the running system. They do not need to summarize the Gist's contents themselves; they trust the tooling. This is a sophisticated interaction pattern: the user acts as a knowledge router, pointing to external sources, while the assistant acts as an executor that can ingest those sources and act on them.

Conclusion

Message 5852 is a masterclass in minimal, high-leverage communication. In three words and a URL, the user resolved a debugging impasse that could have consumed hours or days. The message works because it arrives at the precise moment when the assistant has exhausted its own knowledge — it has tried backends, checked logs, probed code, and found no path forward. The user provides not a hint or a suggestion but a complete, tested solution from someone who has already walked this road. It is a reminder that in complex technical work, the most valuable resource is often not more computation but the right reference at the right time.