Garbage In, Garbage Out: Systematically Testing MoE Backends for Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, the difference between a working configuration and a silent failure often comes down to a single flag. Message [msg 5967] captures one such moment: the assistant has just launched a smoke test of the Qwen3.5-397B-A17B-NVFP4 model using the flashinfer_cutedsl MoE runner backend on an 8× RTX PRO 6000 Blackwell (SM120) system, and the results are unambiguous—and damning. The model outputs nothing but repeated exclamation marks: !!!!!!!!!!!!!!!!. This is the unmistakable signature of a kernel that is executing but producing numerical garbage, a failure mode that is far more dangerous than a crash because it can silently corrupt downstream results.

The Context: A Systematic Backend Hunt

This message sits at the climax of an exhaustive search for a working inference configuration on Blackwell GPUs. The assistant had recently upgraded the entire software stack to nightly builds—PyTorch 2.12.0.dev20260307+cu130, FlashInfer 0.6.5, and the latest SGLang main branch—to support the Qwen3.5-397B-A17B-NVFP4 model, a 397-billion-parameter Mixture-of-Experts (MoE) model quantized to NVFP4 (NVIDIA's 4-bit floating-point format). The challenge was that Blackwell GPUs (compute capability 12.0, or SM120) were so new that many kernel implementations had not yet been validated on them.

The assistant had already established that the default auto backend produced garbage output on SM120. A prior test with flashinfer_trtllm for the MoE runner had crashed because the TRT-LLM fused MoE kernels were compiled only for SM100 (compute capability 10.0) and could not execute on SM120 hardware. The flashinfer_cutlass MoE backend, however, had worked correctly—producing coherent text. But the assistant was not satisfied with a single working configuration; the goal was to find the fastest correct configuration, and flashinfer_cutedsl (CUTE DSL) promised JIT-compiled kernels that could be optimized specifically for the target architecture.

The Message: A Diagnosis and a Pivot

The subject message begins with a bold, emphatic diagnosis:

Garbage output with flashinfer_cutedsl! The !!!! pattern is back. So flashinfer_cutedsl has the same SM120 kernel issue as the default auto backend. Only flashinfer_cutlass works correctly.

This is not merely an observation; it is a reasoned conclusion drawn from a pattern. The assistant has now tested three MoE backends on SM120: auto (garbage), flashinfer_trtllm (crash), flashinfer_cutedsl (garbage), and flashinfer_cutlass (working). The pattern reveals that the issue is not random—it is specific to certain kernel implementations. The flashinfer_cutlass backend, which uses CUTLASS (CUDA Templates for Linear Algebra Subroutines), appears to have the correct SM120 code paths, while the CUTE DSL and auto-selected backends do not.

The reasoning here is diagnostic in nature: the assistant is performing a differential analysis, varying one parameter at a time (the MoE runner backend) while holding all other configuration constant, and observing the output quality. The repeated ! pattern is a known failure signature for FP4 quantization on unsupported architectures—it indicates that the quantized weights are being interpreted incorrectly, producing near-random activations that collapse to a single repeated token.

The Reasoning Behind the Next Step

After delivering the diagnosis, the assistant immediately pivots to the next experiment:

Let me also check flashinfer_trtllm for the dense FP4 GEMM (not MoE) — the --fp4-gemm-backend flag. Let me stop, test with flashinfer_cutlass MoE + flashinfer_trtllm dense FP4:

This reveals a critical piece of reasoning: the assistant has realized that the MoE runner backend and the dense FP4 GEMM backend are independent configuration axes. Even though flashinfer_trtllm failed for the MoE runner (because the fused MoE kernel is SM100-only), it might still work for the dense FP4 GEMM operation (the matrix multiplication in non-MoE layers). The --fp4-gemm-backend flag controls a different kernel path than --moe-runner-backend. By decoupling these two settings, the assistant can potentially use flashinfer_trtllm for the dense GEMM (which might be faster) while using flashinfer_cutlass for the MoE operations (which is known to work).

This is a sophisticated optimization strategy. Rather than settling for the first working configuration, the assistant is systematically exploring the combinatorial space of backend options to find the optimal combination. The assumption is that different kernel implementations have different strengths: TRT-LLM kernels are hand-tuned by NVIDIA for maximum performance on dense operations, while CUTLASS kernels are more portable and robust across architectures. By mixing backends, the assistant hopes to get the best of both worlds.

Assumptions and Knowledge Requirements

To fully understand this message, one must be familiar with several layers of technical context:

  1. The Blackwell architecture (SM120): NVIDIA's RTX PRO 6000 Blackwell GPUs have compute capability 12.0, which is distinct from the Hopper (SM90) and Blackwell (SM100) architectures. Kernel code compiled for SM100 will not run on SM120, and vice versa. This is why flashinfer_trtllm crashed—it was compiled with -arch=sm100 and lacked SM120 code paths.
  2. NVFP4 quantization: NVIDIA's 4-bit floating-point format requires specialized kernel support for matrix multiplication. The mm_fp4 function in FlashInfer dispatches to different backends (cudnn, trtllm, cutlass, cute-dsl) depending on availability and user preference. Not all backends support all architectures.
  3. The MoE runner architecture: SGLang's MoE implementation has multiple runner backends that handle the complex routing and computation for mixture-of-experts layers. The flashinfer_cutlass backend uses CUTLASS templates, flashinfer_cutedsl uses NVIDIA's CUTE DSL, and flashinfer_trtllm uses TensorRT-LLM kernels.
  4. The garbage output signature: Repeated ! characters are a telltale sign of FP4 quantization failure. When the kernel misinterprets the quantized weight format, the resulting activations are effectively random noise, which the language model's softmax layer collapses into a single high-probability token. The assistant's assumptions in this message are reasonable and grounded in prior evidence. The key assumption is that the flashinfer_trtllm dense FP4 GEMM backend might work on SM120 even though the flashinfer_trtllm MoE runner failed, because they use different kernel code paths. This is a well-motivated hypothesis: the dense GEMM operation is simpler than the fused MoE kernel and may have broader architecture support.

The Thinking Process Visible in the Message

The message reveals a clear chain of reasoning:

  1. Observation: The smoke test produced garbage output (!!!!!!!!!!!!!!!!), confirming that flashinfer_cutedsl has the same SM120 issue as the default auto backend.
  2. Pattern recognition: The assistant has now tested multiple backends and identified a consistent pattern—only flashinfer_cutlass works correctly on SM120 for MoE operations.
  3. Hypothesis formation: The dense FP4 GEMM backend (--fp4-gemm-backend) is a separate code path from the MoE runner backend. Even though flashinfer_trtllm failed for MoE, it might work for dense GEMM.
  4. Action planning: The assistant formulates a concrete next step: stop the current server, clean up GPU memory, and launch a new test with flashinfer_cutlass for MoE and flashinfer_trtllm for dense FP4 GEMM. The thinking is methodical and scientific. Each experiment isolates a single variable, the results are interpreted against a known failure signature, and the conclusions directly inform the next experiment. This is not random trial-and-error; it is a systematic search of the configuration space guided by architectural knowledge.

Output Knowledge Created

This message creates several pieces of valuable knowledge:

  1. flashinfer_cutedsl is incompatible with SM120 for MoE: This is a concrete, tested finding. Future deployments on Blackwell GPUs should avoid this backend for MoE operations.
  2. flashinfer_cutlass is the only confirmed working MoE backend on SM120: This establishes a baseline correct configuration.
  3. The dense FP4 GEMM backend is worth testing independently: The assistant identifies that --fp4-gemm-backend and --moe-runner-backend control separate code paths, opening up a combinatorial optimization opportunity.
  4. A test methodology for backend validation: The smoke test script (smoke_test_qwen.py) with its garbage detection assertion (assert "!" * 10 not in resp) provides a reliable way to detect FP4 kernel failures automatically.

Broader Significance

This message exemplifies a fundamental challenge in deploying AI models on new hardware: the gap between what should work (based on documentation and architecture support) and what actually works (based on empirical testing). The Blackwell GPUs in this system are cutting-edge hardware, and the software ecosystem is still catching up. Kernel implementations that were written and tested on SM100 may silently produce incorrect results on SM120 without any error messages—only garbage output.

The assistant's response to this challenge is instructive: rather than giving up or blindly accepting the first working configuration, it systematically probes the boundaries of the configuration space, building a mental map of which combinations work and which fail. This is the kind of rigorous, evidence-driven debugging that separates production-quality deployments from fragile prototypes. The message [msg 5967] is a small but crucial step in that process—a diagnosis that narrows the search space and sets the stage for the next experiment.