The Pivot Point: Investigating FlashInfer TRT-LLM Backend Support on Blackwell SM120

In the high-stakes world of deploying large language models on cutting-edge hardware, every tok/s matters. When the user running an 8× RTX PRO 6000 Blackwell (SM120) server saw only ~72 tok/s from their Qwen3.5-397B-A17B-NVFP4 deployment—a number they described as "really low"—they immediately asked two sharp questions: does the model support built-in Multi-Token Prediction (MTP), and can the GEMM backends be tuned for the SM120 architecture? Message 5947 is the critical inflection point where the assistant transitions from research into active backend testing, setting the stage for the exhaustive trial that would ultimately determine the production configuration.

The Message

The assistant wrote:

Now let me check if flashinfer_trtllm for dense FP4 GEMM works on SM120, and benchmark the three backends. Let me also look at the MoE runner options:

>

``bash ssh root@10.1.230.174 'grep -n "is_sm120\|is_sm100\|is_blackwell\|compute_cap\|major.*12\|major.*10" /root/sglang-main/python/sglang/srt/layers/moe/moe_runner/ -r 2>&1 | head -20' ``

>

Results: `` grep: .../flashinfer_trtllm.cpython-312.pyc: binary file matches /root/sglang-main/python/sglang/srt/layers/moe/moe_runner/flashinfer_trtllm.py:28: is_sm120_supported, /root/sglang-main/python/sglang/srt/layers/moe/moe_runner/flashinfer_trtllm.py:38:if is_flashinfer_available() and is_sm120_supported(): ``

>

``bash ssh root@10.1.230.174 'ls /root/sglang-main/python/sglang/srt/layers/moe/moe_runner/' ``

>

Results: `` __init__.py __pycache__ base.py deep_gemm.py flashinfer_trtllm.py marlin.py runner.py triton.py triton_kernels.py ``

At first glance, this appears to be a routine investigation. But this message represents a pivotal moment: the assistant has just completed an extensive research phase (messages 5929–5946) where it discovered that the NVFP4 checkpoint does contain MTP weights (1553 tensors including all 512 experts), found that SGLang has a dedicated qwen3_5_mtp.py model file, and learned that --speculative-algorithm NEXTN is the mechanism to enable it. Now, with that knowledge in hand, the assistant turns to the second half of the user's question: can the GEMM backends be improved?

Why This Message Was Written: The Reasoning and Motivation

The motivation behind message 5947 is rooted in a fundamental tension in ML engineering: the gap between theoretical peak FLOPS and achieved throughput. The RTX PRO 6000 Blackwell GPUs are architectural marvels with FP4 tensor core support, but the software stack to exploit them is still maturing. The assistant had deployed the model with --moe-runner-backend flashinfer_cutlass and --fp4-gemm-backend flashinfer_cudnn—a configuration that worked but delivered only ~72 tok/s.

The user's question about "tuning gemms for SM120" implicitly challenged the assistant's default choices. The assistant needed to determine: is flashinfer_cutlass actually the best option for SM120, or could flashinfer_trtllm—which uses NVIDIA's TensorRT-LLM hand-tuned kernels—deliver significantly better performance? The TRT-LLM backend is known for fusing the entire MoE computation (routing, GEMM1, activation, GEMM2) into a single kernel, which can dramatically reduce launch overhead and improve throughput.

But before benchmarking, the assistant needed to verify a prerequisite: does the flashinfer_trtllm MoE runner even support SM120? The Blackwell RTX PRO 6000 has compute capability 12.0, which is distinct from the B200's SM100 architecture. SGLang's codebase explicitly distinguishes between is_sm100_supported() (for B200, compute capability 10.x) and is_sm120_supported() (for Blackwell consumer/ workstation GPUs like the RTX PRO 6000). If the TRT-LLM backend only supported SM100, it would crash on SM120—and the assistant wisely chose to check the source code rather than waste time on a trial that would fail.

How Decisions Were Made

The assistant's decision-making in this message reveals a methodical, source-first investigative approach. Rather than blindly launching a server with --moe-runner-backend flashinfer_trtllm and waiting for it to crash, the assistant:

1. Grepped the MoE runner directory for architecture-specific guards (is_sm120, is_sm100, is_blackwell, compute_cap, major.*12, major.*10). This targeted search immediately surfaced the relevant code paths.
2. Listed the available runner files to understand the full landscape of MoE backends. The listing reveals seven options: base.py (abstract interface), deep_gemm.py, flashinfer_trtllm.py, marlin.py, runner.py (dispatch logic), triton.py, and triton_kernels.py. Notably absent are dedicated flashinfer_cutlass.py or flashinfer_cutedsl.py files—these backends are handled differently, as fused functions within the MoE layer itself rather than standalone runner cores. The grep results were immediately revealing. Line 28 of flashinfer_trtllm.py imports is_sm120_supported, and line 38 wraps the entire backend in a conditional: if is_flashinfer_available() and is_sm120_supported():. This confirms that the TRT-LLM MoE runner was explicitly designed to support SM120—it's not an afterthought or accidental compatibility. The presence of a compiled .pyc file (the binary match) confirms the code has been imported and executed at least once, validating the import chain works.

Assumptions Made by the Assistant

Several assumptions underpin this message, some explicit and some implicit:

The primary assumption is that flashinfer_trtllm would be faster than flashinfer_cutlass on SM120. This is a reasonable engineering judgment: TRT-LLM kernels are hand-tuned by NVIDIA's kernel engineers, typically fusing multiple operations into a single launch, while Cutlass kernels are more generic template-based implementations. However, this assumption would later prove incorrect—when the assistant actually tested flashinfer_trtllm (in subsequent messages), the server crashed during model loading. The TRT-LLM backend, despite its SM120 support guard, was incompatible with the specific FP4 weight layout or routing configuration of the Qwen3.5 model.

The assistant also assumed that checking source code was the most efficient first step. This reflects a deep understanding of the SGLang codebase architecture: the MoE runner directory contains the backend implementations, and architecture-specific guards are the canonical way SGLang handles SM120 vs SM100 differentiation. A less experienced engineer might have simply tried launching the server and observed the error, but the assistant's approach saved time by pre-validating compatibility.

An implicit assumption is that the three backends worth benchmarking are flashinfer_trtllm, flashinfer_cutlass, and flashinfer_cutedsl (the "three backends" referenced in the message). The assistant had already learned from the earlier research task that flashinfer_cudnn was the working FP4 GEMM backend, and that flashinfer_cutedsl used CUTE DSL for JIT-compiled kernels. The assumption that these three MoE backends are the primary candidates is reasonable, but it overlooks the possibility that none of them would work correctly—which is exactly what happened.

Input Knowledge Required

To understand this message, one needs substantial context about the SGLang inference engine architecture:

• MoE runner backends: SGLang supports multiple backends for Mixture-of-Experts computation, each implementing the same interface (MoeRunnerCore) but using different kernel technologies. flashinfer_trtllm uses NVIDIA's TensorRT-LLM kernels, flashinfer_cutlass uses the Cutlass template library, flashinfer_cutedsl uses CUTE DSL, and triton uses OpenAI's Triton language.
• SM120 vs SM100: NVIDIA's Blackwell architecture has two distinct compute capability families. The B200/B100 datacenter GPUs have SM100 (compute capability 10.x), while the RTX PRO 6000 workstation GPU has SM120 (compute capability 12.0). These require different kernel compilations and have different feature support.
• FP4 quantization: The Qwen3.5-397B-A17B-NVFP4 model uses NVIDIA's FP4 quantization format (NVFP4), which packs two 4-bit values into each byte. This requires specialized GEMM kernels that understand the NVFP4 format, including block scaling factors and the 128×4 scale factor layout.
• The SGLang codebase structure: The MoE runner implementations live in /sglang/srt/layers/moe/moe_runner/, and architecture guards like is_sm120_supported() are imported from shared utility modules.

Output Knowledge Created

This message produced two concrete pieces of knowledge:

1. Confirmation that flashinfer_trtllm explicitly supports SM120: The import of is_sm120_supported and the conditional guard on line 38 prove that the TRT-LLM MoE backend was designed with SM120 compatibility in mind. This is non-trivial—it means the backend has been tested (or at least compiled) for compute capability 12.0, and the kernel implementations include SM120-specific code paths.
2. The full inventory of available MoE runner backends: The directory listing reveals the complete set of options. Notably, flashinfer_cutlass and flashinfer_cutedsl are not present as standalone runner files—they are handled as fused functions within the FusedMoE layer and the StandardTokenDispatcher. This architectural insight is crucial: it means switching between flashinfer_trtllm and flashinfer_cutlass involves fundamentally different code paths, not just swapping a runner core.

Mistakes and Incorrect Assumptions

The most significant mistake in this message is not what it says, but what it doesn't yet know. The assistant assumes that because flashinfer_trtllm has SM120 support guards, it will work correctly on the RTX PRO 6000. In reality, when the assistant attempts to launch the server with --moe-runner-backend flashinfer_trtllm in message 5960, the process is killed immediately (exit code 137, SIGKILL from OOM or assertion failure). The assistant then tries flashinfer_cutedsl in message 5975, which produces NaN outputs. Only flashinfer_cutlass for MoE combined with flashinfer_cudnn for FP4 GEMM produces correct results.

The root cause of the TRT-LLM failure is never fully diagnosed, but it likely relates to weight layout requirements. The TRT-LLM backend expects weights in a specific shuffled format optimized for its fused kernel, and the NVFP4 checkpoint may not have been pre-shuffled for this layout. The flashinfer_cutlass backend, being more flexible, can handle the unshuffled weights directly.

Additionally, the assistant's framing of "benchmark the three backends" implicitly assumes that all three will work and the task is simply to measure which is fastest. In reality, the task becomes a compatibility sieve: test each backend, eliminate the ones that crash or produce garbage, and use whatever remains. This is a common pattern in early-adopter ML deployments where the software stack hasn't fully stabilized for new hardware.

The Thinking Process

The reasoning visible in this message reveals a systematic, layered approach to problem-solving. The assistant has just completed an extensive research phase (messages 5929–5946) that answered the user's first question about MTP. Now, in message 5947, it pivots to the second question about GEMM tuning.

The thinking proceeds in two parallel tracks:

• Track 1: Check if flashinfer_trtllm for dense FP4 GEMM works on SM120. This is about the FP4 GEMM backend (--fp4-gemm-backend), which handles the dense (non-MoE) FP4 matrix multiplications.
• Track 2: Look at the MoE runner options. This is about the MoE backend (--moe-runner-backend), which handles the routed expert computations. The assistant uses grep to search for architecture-specific guards because this is the fastest way to determine compatibility without launching a server. The choice of search terms (is_sm120, is_sm100, is_blackwell, compute_cap, major.*12, major.*10) is carefully designed to catch all possible architecture guards—both the explicit SM120 checks and any broader Blackwell checks that might encompass SM120. The directory listing serves a different purpose: it reveals the architectural landscape. The absence of flashinfer_cutlass.py and flashinfer_cutedsl.py as standalone files is a significant discovery that shapes the assistant's subsequent investigation. In message 5951, the assistant follows up by searching for where these backends are registered, finding them in flashinfer_cutedsl_moe.py and the StandardTokenDispatcher.

The Broader Significance

Message 5947 sits at the exact midpoint of a multi-stage optimization journey. The assistant has already:

1. Upgraded the entire stack to nightly builds (PyTorch 2.12.0, flashinfer 0.6.5, SGLang main)
2. Built sgl-kernel from source with SM120 FP4 support
3. Confirmed MTP exists in the checkpoint And will soon:
4. Test flashinfer_trtllm (crashes)
5. Test flashinfer_cutedsl (NaN outputs)
6. Settle on flashinfer_cutlass + flashinfer_cudnn as the only working combo
7. Discover the FP8 KV cache accuracy issue and force BF16
8. Achieve ~172 tok/s single-request and >2100 tok/s at high concurrency This message is the gateway to the testing phase. The assistant's methodical, source-first approach—checking code before running experiments—is what separates professional ML engineering from trial-and-error hacking. It's a small message with large consequences: the decision to investigate flashinfer_trtllm leads to a dead end, but the systematic exploration of all backends ultimately produces a stable, production-quality configuration. The assistant's willingness to test, fail, and pivot is the real story here, and message 5947 is where that story begins.