The Silence Before the Storm: A Pivotal Log Check in the Search for a Working Blackwell MoE Backend

In the high-stakes world of deploying large language models on cutting-edge hardware, a single line of log output can represent either a breakthrough or a dead end. Message [msg 5964] in this opencode session captures one such moment — a brief, almost mundane bash command that checks the tail of a server log file. But within the context of the session's broader arc, this message is a quiet pivot point: the moment when a promising new approach (flashinfer_cutedsl) appears to succeed, only for that success to later unravel into yet another failure.

The Message

The message itself is deceptively simple:

[assistant] [bash] ssh root@10.1.230.174 'tail -30 /tmp/sglang_cutedsl_test.log' [2026-03-07 15:13:29 TP0] GDN kernel dispatcher: decode=TritonGDNKernel, extend=TritonGDNKernel, verify=TritonGDNKernel [2026-03-07 15:13:29 TP5] Capture cuda graph begin. This can take up to several minutes. avail mem=19.85 GB [2026-03-07 15:13:29 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=19.85 GB [2026-03-07 15:13:29 TP0] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, ...

The assistant issues a remote SSH command to inspect the server log for a test deployment of the Qwen3.5-397B-A17B-NVFP4 model using the flashinfer_cutedsl MoE runner backend. The log output shows that the server has loaded successfully: the GDN (Global Dispatch Network) kernel dispatcher has initialized with Triton-based kernels for decode, extend, and verify phases, and CUDA graph capture has begun on two tensor-parallel ranks (TP0 and TP5). The available memory per GPU is reported at 19.85 GB, and the graph capture is iterating through a batch size schedule starting at 1 and progressing through powers of two and intermediate values.

Why This Message Was Written

To understand why this message exists, one must trace the assistant's reasoning through the preceding messages. The session had reached a critical juncture in deploying the Qwen3.5-397B-A17B-NVFP4 model — a 397-billion-parameter Mixture-of-Experts (MoE) model quantized with NVIDIA's NVFP4 format — on a machine equipped with eight RTX PRO 6000 Blackwell GPUs (compute capability SM120). The assistant had already upgraded the entire software stack to nightly builds: PyTorch 2.12.0.dev20260307+cu130, FlashInfer 0.6.5, and the latest SGLang main branch. The sgl-kernel package had been built from source with SM120 support by applying patches for CMake policy guards, CUDA 13 cccl include paths, and FlashAttention-3 fallback.

But the critical question remained: which backend configuration would produce correct output on SM120? The assistant had been systematically testing each combination of MoE runner backend and FP4 GEMM backend. The first candidate, flashinfer_trtllm, had failed catastrophically in [msg 5960] — the server process was killed immediately, and the logs revealed that trtllm_fp4_block_scale_moe attempted to compile for SM100 (compute capability 10.x), which is incompatible with the SM120 Blackwell GPUs. The fused TRT-LLM MoE kernels were simply not compiled for the correct architecture.

This led the assistant to formulate a new hypothesis in [msg 5962]: "Let me try flashinfer_cutedsl instead — it should JIT-compile for SM120." The reasoning was sound. Unlike the TRT-LLM kernels, which are pre-compiled for specific architectures, the CUTE DSL (Domain-Specific Language) kernels use a Just-In-Time compilation approach. They generate CUDA code at runtime and compile it for the exact target architecture. On SM120 Blackwell GPUs, this should theoretically produce working kernels. The assistant launched the server with --moe-runner-backend flashinfer_cutedsl in [msg 5963], and the command timed out after 120 seconds — a positive sign, since it meant the server hadn't crashed immediately but was likely still initializing.

Message [msg 5964] is the follow-up: the assistant checks the log to see whether the server actually started successfully. The log output confirms that it did.## The Reasoning and Assumptions

The assistant's decision to try flashinfer_cutedsl was grounded in a clear technical understanding of the CUDA compilation landscape. The key assumption was that CUTE DSL kernels, being JIT-compiled, would adapt to the SM120 architecture where pre-compiled TRT-LLM kernels could not. This is a reasonable assumption in theory: JIT compilation from a high-level DSL should target whatever GPU is present. However, the assumption carried hidden risks. The CUTE DSL kernel templates may contain architecture-specific code paths, tensor layouts, or warp-level primitives that are only validated for SM90 (Hopper) or SM100 (Blackwell 100-series). On SM120, these templates might compile successfully but produce incorrect numerical results due to subtle differences in the hardware instruction set or memory subsystem.

The assistant also assumed that the flashinfer_cudnn FP4 GEMM backend (used for the dense layers) would remain compatible regardless of the MoE backend choice. This was a safe assumption given that cudnn is a mature, well-tested library with explicit SM120 support in CUDA 13.

What the Log Output Actually Tells Us

The log output in message [msg 5964] reveals several important details:

The GDN kernel dispatcher has selected TritonGDNKernel for all three phases (decode, extend, verify). This is the expected Triton-based attention backend, confirming that the --attention-backend triton flag is working correctly.
CUDA graph capture has begun on tensor-parallel ranks TP0 and TP5. The fact that graph capture is proceeding — and that it reports 19.85 GB of available memory — indicates that the model weights have been loaded successfully and the initial forward pass through the MoE layers using flashinfer_cutedsl did not crash. This is a significant milestone: it means the CUTE DSL kernels compiled and ran without an immediate runtime error.
The batch size schedule shows the server is preparing optimized CUDA graphs for batch sizes 1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184... This is the standard SGLang graph capture process, which pre-compiles CUDA graphs for common batch sizes to reduce launch latency during inference. The log is truncated (note the trailing ...), so we don't see the full batch size range or the completion of graph capture. But the server was clearly alive and making progress — a stark contrast to the immediate crash with flashinfer_trtllm.

The Bitter Aftermath

What makes message [msg 5964] so poignant is what comes next. In [msg 5965], the assistant confirms the server is healthy and begins a smoke test. In [msg 5966], the smoke test reveals garbage output: the model produces repeated exclamation marks (!!!!!!!!!!!!!!!!), triggering an assertion failure in the test script. The flashinfer_cutedsl backend, despite loading successfully and passing CUDA graph capture, produces numerically incorrect results on SM120.

The assistant's reaction in [msg 5967] is telling: "Garbage output with flashinfer_cutedsl! The !!!! pattern is back. So flashinfer_cutedsl has the same SM120 kernel issue as the default auto backend. Only flashinfer_cutlass works correctly."

This reveals a crucial insight: a backend can load, compile, and even run without crashing, yet still produce completely wrong outputs. The CUTE DSL kernels compiled successfully for SM120 — no runtime errors, no memory violations — but the numerical computation was silently corrupted. This is the most dangerous class of failure in GPU kernel development: silent correctness bugs that manifest only in the quality of the model's output.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of CUDA compute capabilities: SM120 is the compute capability for NVIDIA Blackwell GPUs (RTX PRO 6000). SM100 is the previous Blackwell generation. Different compute capabilities require different kernel compilations.
Understanding of MoE inference architecture: The Mixture-of-Experts layers in models like Qwen3.5-397B require specialized fused kernels that combine routing, matrix multiplication, activation functions, and output combination. The MoE runner backend controls which kernel implementation is used.
Familiarity with JIT compilation concepts: The CUTE DSL approach generates CUDA code at runtime, theoretically adapting to the target architecture. The TRT-LLM approach uses pre-compiled kernels for specific architectures.
Knowledge of the SGLang server architecture: The log output references GDN kernel dispatchers, tensor-parallel ranks, CUDA graph capture, and batch size schedules — all SGLang-specific concepts.## Output Knowledge Created This message, combined with the subsequent test results, creates valuable knowledge for the session: 1. flashinfer_cutedsl loads successfully on SM120 — it does not crash at startup, unlike flashinfer_trtllm. This is a positive data point for future Blackwell development. 2. Successful loading does not imply correct computation — the CUTE DSL kernels produce silent numerical corruption on SM120. This is a critical warning for anyone attempting to deploy on Blackwell GPUs. 3. The only working MoE backend for SM120 is flashinfer_cutlass — this becomes the definitive answer after exhaustive testing of all available options. 4. The flashinfer_cudnn FP4 GEMM backend is compatible with all MoE backends — it can be paired with any MoE runner without issues.

The Thinking Process

The assistant's reasoning in the messages leading up to [msg 5964] reveals a methodical, hypothesis-driven approach to debugging. After the flashinfer_trtllm failure, the assistant didn't just try random alternatives. It reasoned about why TRT-LLM failed (pre-compiled for SM100, not SM120) and used that insight to select flashinfer_cutedsl as the next candidate (JIT-compiles for the target architecture). This is a textbook example of using failure analysis to guide the next experiment.

The assistant also demonstrated good engineering judgment by checking the log before running the smoke test. Rather than assuming the server was healthy based on the timeout, it verified the actual startup state. This prevented a false positive — if the assistant had immediately run the smoke test without checking the log, it might have attributed a failure to a server that hadn't finished loading.

Broader Implications

This message and its surrounding context illustrate a fundamental challenge in deploying large language models on new hardware architectures. The Blackwell SM120 GPU represents a significant architectural shift from previous generations, and the software ecosystem is still catching up. Kernel libraries like FlashInfer must be recompiled and validated for each new compute capability, and even then, JIT-compiled kernels may produce incorrect results that are difficult to detect without rigorous numerical validation.

The session's ultimate resolution — settling on flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM — represents a pragmatic compromise. The cutlass backend may not be the fastest option theoretically, but it is the only option that produces correct results on SM120. In production machine learning, correctness always trumps performance.

Message [msg 5964] is, in retrospect, the calm before the storm. The log output shows a server that appears to be working perfectly — CUDA graphs are being captured, memory is available, the kernel dispatcher is initialized. Everything looks right. But beneath the surface, the CUTE DSL kernels are silently producing corrupted arithmetic, and it will take a smoke test to reveal the truth. It is a reminder that in the world of GPU kernel development, a successful launch is only the beginning of the validation journey.