The Last Backend Standing: Tracing the Systematic Elimination of FlashInfer TRT-LLM on Blackwell SM120

In the high-stakes world of deploying large language models on cutting-edge hardware, few tasks are as methodical—and as unforgiving—as finding a working kernel backend for a new GPU architecture. Message [msg 5970] captures a pivotal moment in this process: the assistant reads the crash log from a failed attempt to use the flashinfer_trtllm backend for dense FP4 matrix multiplication on an NVIDIA Blackwell RTX PRO 6000 GPU (compute capability SM120). This single bash command and its truncated traceback output represent the culmination of an exhaustive backend elimination process that systematically narrowed the field of viable configurations from five candidates down to just two.

The Context: A Systematic Backend Search

The message sits within a larger investigation that began several rounds earlier. The assistant had been tasked with deploying the Qwen3.5-397B-A17B-NVFP4 model—a massive 397-billion-parameter Mixture-of-Experts (MoE) model quantized to NVIDIA's FP4 format—on an 8-GPU Blackwell machine. The challenge was that Blackwell (SM120, compute capability 12.0) is a brand-new architecture, and many kernel backends in the FlashInfer and SGLang ecosystem had not yet been validated for it.

The assistant had already discovered, through a sequence of careful experiments, that:

The Experiment That Failed

In [msg 5969], the assistant launched a server with a carefully constructed configuration:

--moe-runner-backend flashinfer_cutlass \
--fp4-gemm-backend flashinfer_trtllm

This was a deliberate hybrid approach: keep the working MoE backend (cutlass) while testing a potentially faster dense GEMM backend (trtllm). The server was killed immediately—the shell reported PID: 48326 Killed—indicating a fatal error during model loading or the first forward pass.

Message [msg 5970] is the assistant's response: it reads the log file to understand why. The output shows a Python traceback threading through the model's forward pass:

File "torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
File "sglang/srt/layers/linear.py", line 453, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
File "sglang/srt/layers/quantization/modelopt_quant.py", line 1308, in ...

The traceback is truncated, but the critical information is the path: it crashes in linear.py's forward method, which calls self.quant_method.apply(). This is the dense linear layer attempting to use the FP4 quantization method with the flashinfer_trtllm backend, and failing. The crash occurs during model initialization or the very first inference step—before the server could even become healthy.

What This Message Reveals

This message is significant for several reasons. First, it confirms that flashinfer_trtllm is incompatible with SM120 in both contexts—both for MoE and for dense GEMM. The TRT-LLM kernels, which are hand-tuned NVIDIA kernels typically offering the best performance, were compiled only for SM100 (compute capability 10.x) and lack SM120 code paths. This is a fundamental architectural limitation, not a configuration issue.

Second, the message demonstrates the assistant's disciplined debugging methodology. Rather than guessing or speculating, the assistant immediately reads the log file to gather evidence. The truncated traceback, while incomplete, provides enough information to pinpoint the failure location. The assistant's next message ([msg 5971]) confirms the conclusion: "Confirmed: trtllm backend for dense FP4 also doesn't support SM120 (capability 120). Only cutlass and cudnn work for dense FP4 on SM120."

Third, this message represents a critical narrowing of the search space. After testing five backends across two dimensions (MoE runner and FP4 GEMM), the viable set is now:

| Backend | MoE Runner | Dense FP4 GEMM | |---------|-----------|----------------| | flashinfer_cutlass | ✅ Works | ✅ Works | | flashinfer_cudnn | ❌ Not an option | ✅ Works | | flashinfer_trtllm | ❌ SM120 crash | ❌ SM120 crash | | flashinfer_cutedsl | ❌ Garbage output | ❌ Not tested |

The only fully working combination is flashinfer_cutlass for MoE with either flashinfer_cudnn or flashinfer_cutlass for dense FP4 GEMM.

Assumptions and Knowledge Required

To fully understand this message, one needs knowledge of several layers of the ML inference stack. The reader must understand what a "GEMM backend" is—the kernel implementation that performs general matrix-matrix multiplication for the FP4 quantized format. They must know about NVIDIA's compute capability numbering (SM100 = Hopper/H100, SM120 = Blackwell/RTX PRO 6000) and why kernel code compiled for one architecture may not run on another. They need familiarity with SGLang's server architecture, particularly the distinction between MoE runner backends (which handle the expert-parallel MoE computation) and FP4 GEMM backends (which handle dense linear layers). And they need to understand the quantization scheme: modelopt_fp4 is NVIDIA's ModelOpt FP4 format, which requires specialized kernels that support the unusual 4-bit block-scaled format.

The assistant also makes an implicit assumption worth examining: that the crash is due to architecture incompatibility rather than a configuration error or library version mismatch. This assumption is reasonable given the pattern of failures—every attempt to use trtllm-based backends on SM120 has failed in the same way—but it's worth noting that the traceback alone doesn't definitively prove the root cause. The assistant's conclusion in the next message treats this as confirmed, and the subsequent pivot to testing flashinfer_cutlass for both MoE and dense GEMM validates that judgment.

Output Knowledge Created

This message creates concrete, actionable knowledge: the flashinfer_trtllm backend cannot be used for dense FP4 GEMM on Blackwell SM120 GPUs. This is a negative result, but in engineering contexts, negative results are often as valuable as positive ones. Knowing what doesn't work prevents wasted effort and guides future optimization attempts toward viable paths.

The message also contributes to the broader knowledge base about Blackwell compatibility. As one of the early deployments on SM120 hardware, each backend test adds to the collective understanding of which SGLang and FlashInfer features work on this architecture. The assistant's systematic documentation of failures—complete with log excerpts and tracebacks—creates a reference that could help other engineers facing similar issues.

The Thinking Process

The reasoning visible in this message and its surrounding context is methodical and hypothesis-driven. The assistant is working through a decision tree: for each backend combination, launch the server, check if it starts, run a smoke test, and evaluate correctness and performance. When a launch fails, the first step is always to read the logs. When a launch succeeds but produces garbage (as with flashinfer_cutedsl), the smoke test catches it.

This approach reflects a deep understanding of the failure modes in GPU kernel development. Crashes during initialization typically indicate missing or incompatible kernel code paths. Garbage output (the "!!!!" pattern) suggests a kernel that runs without errors but computes incorrectly—often due to incorrect scale factor handling or data layout assumptions. Both failure modes are distinct and require different responses.

Conclusion

Message [msg 5970] is a small but crucial piece of a larger puzzle. In the span of a single bash command and a truncated traceback, it eliminates one of the five candidate backends and narrows the path forward. The assistant's disciplined approach—test, fail, read logs, conclude, pivot—is a textbook example of systematic debugging in the challenging domain of ML infrastructure. The final production configuration would settle on flashinfer_cutlass for both MoE and dense FP4 GEMM, achieving ~172 tok/s at single-request concurrency and over 2100 tok/s at high concurrency. But that success was built on a foundation of careful elimination, one backend at a time.