The Pivot to GEMM Tuning: Investigating FP4 Backend Options for Blackwell SM120

In the sprawling, multi-session effort to deploy Qwen3.5-397B-A17B-NVFP4 on an 8× RTX PRO 6000 Blackwell machine, message [msg 5942] marks a critical inflection point. The assistant had just confirmed that the model's NVFP4 checkpoint contained MTP (Multi-Token Prediction) heads and that SGLang's --speculative-algorithm NEXTN could leverage them. But the user's question—"Is the backend the fastest one we can get?"—lingered. This message is the assistant's pivot from verifying MTP feasibility to investigating GEMM backend performance, a shift that reveals how the assistant thinks about optimization: systematically confirm feature support, then drill into performance tuning.

The Immediate Context: Two Questions, One Answered

The user's previous message ([msg 5928]) posed two sharp questions: "Is there built-in MTP?" and "Can we tune GEMMs for SM120? Is the backend the fastest one we can get?" The assistant had spent messages [msg 5929] through [msg 5941] thoroughly answering the first question. It discovered that the NVFP4 checkpoint indeed contains MTP layers—1,553 weight tensors including a full set of 512 experts for the draft module—and that SGLang has a dedicated qwen3_5_mtp.py model file. It traced the auto-detection logic in model_config.py where, when is_draft_model=True and the architecture is Qwen3_5MoeForConditionalGeneration, the model is redirected to Qwen3_5ForCausalLMMTP with num_nextn_predict_layers=1. It also uncovered a critical detail: the MTP layers run in BF16, not FP4, meaning the draft model's MoE layers would use a different precision path than the main model.

By message [msg 5942], the assistant has absorbed all this and is ready to tackle the second question. But before diving into backend experimentation, it pauses to reason about one subtle consequence of the MTP architecture: the speculative MoE runner backend selection.

The Reasoning: Tracing Auto-Selection Logic

The message opens with a reasoning paragraph that demonstrates the assistant's mental model of SGLang's configuration system:

"Good — the speculative moe runner backend auto-selects to auto when main is flashinfer_trtllm, or inherits from the main backend otherwise. Since we're using flashinfer_cutlass, the draft MoE will also use flashinfer_cutlass."

This reasoning is grounded in code the assistant read in message [msg 5941], where it examined lines 2600-2620 of server_args.py. The logic there is nuanced: when the main MoE runner backend is flashinfer_trtllm, the speculative MoE runner backend defaults to auto (because trtllm doesn't support BF16 MoE routing methods needed for draft models). But when the main backend is anything else—like flashinfer_cutlass, which the assistant had already selected as the working configuration—the speculative backend inherits the same value.

This is a moment of synthesis. The assistant is connecting two pieces of knowledge: (1) the MTP draft model runs in BF16, and (2) the speculative MoE backend selection logic. The conclusion is reassuring—there's no hidden incompatibility. The draft model's MoE will use the same flashinfer_cutlass backend as the main model, which is already proven to work on SM120.

The Investigation: Probing FP4 GEMM Backend Options

Having satisfied itself about the speculative backend, the assistant immediately pivots to the core performance question. It formulates a bash command to grep for backend options in fp4_utils.py:

grep -n "flashinfer_trtllm\|trtllm.*fp4\|is_sm120\|is_sm100\|is_blackwell" /root/sglang-main/python/sglang/srt/layers/quantization/fp4_utils.py

The choice of search terms is revealing. The assistant is looking for three things:

flashinfer_trtllm — the alternative FP4 GEMM backend mentioned in the research task ([msg 5930])
trtllm.*fp4 — any FP4-specific code paths within the trtllm backend
is_sm120, is_sm100, is_blackwell — architecture-specific guards that might enable or disable certain backends on Blackwell This grep pattern shows the assistant's mental model of how backend selection works in SGLang. It knows that different GPU architectures may have different backend support, and it's specifically looking for SM120 (compute capability 12.0, the Blackwell RTX PRO 6000) handling. The assistant is hunting for any architecture-specific code that might affect which backends are available or optimal.

Assumptions and Knowledge

The message rests on several assumptions. First, the assistant assumes that flashinfer_trtllm is a viable alternative backend worth testing. This assumption came from the research task ([msg 5930]) which identified it as an option. However, the assistant doesn't yet know whether it will actually work on SM120—that's what the grep is designed to discover.

Second, the assistant assumes that backend selection is primarily a matter of correctness (does it produce valid output?) rather than just performance. This is implicit in the careful approach: the assistant could have simply tried flashinfer_trtllm and measured throughput, but instead it first checks for architecture-specific support. This reflects a lesson learned earlier in the session, where the flashinfer_trtllm and flashinfer_cutedsl backends were found to crash or produce NaN/garbage on SM120 (as noted in the chunk summary).

Third, the assistant assumes that the speculative MoE backend selection logic it traced is correct and up-to-date. This is a reasonable assumption given it's reading the latest main branch code, but it's worth noting that the assistant doesn't verify by actually running the server with MTP enabled—it only checks the code paths.

The input knowledge required to understand this message is substantial. One must know:

The architecture of SGLang's speculative decoding system (EAGLE, NEXTN, etc.)
The distinction between main model and draft model MoE backends
The FP4 quantization format and its GEMM backend options
The Blackwell GPU architecture (SM120 vs SM100) and why it matters
The history of backend failures on SM120 earlier in the session

Output Knowledge Created

This message creates several pieces of output knowledge. First, it confirms that the speculative MoE backend will be flashinfer_cutlass for the draft model, matching the main model. This is important because if the assistant later tries to enable MTP speculative decoding, it won't encounter a backend mismatch.

Second, the grep command will reveal whether flashinfer_trtllm has any SM120-specific handling in fp4_utils.py. If the grep finds is_sm120 guards, that tells the assistant that the backend was designed with SM120 in mind. If it finds nothing, it suggests the backend may not have been tested on Blackwell consumer GPUs.

Third, the message sets up the next phase of investigation. The assistant is gathering data before making a decision about which backend to test. This is characteristic of the session's methodology: gather information, reason about implications, then execute experiments.

The Thinking Process

The visible thinking process in this message is concise but revealing. The assistant starts with a conclusion ("Good") that signals it has processed the speculative backend information and found no issues. It then articulates the auto-selection logic in its own words, demonstrating comprehension. Finally, it announces the next investigation target ("Now let me also check what --fp4-gemm-backend options we should try") and executes the grep.

What's notable is what's not said. The assistant doesn't explicitly state why it's checking flashinfer_trtllm specifically, or why it's looking for SM120 guards. It trusts the reader (and itself) to understand the connection: the research task identified flashinfer_trtllm as an alternative, and the SM120 guards will tell if it's supported. This is efficient communication for an expert audience, but it leaves implicit reasoning that a less experienced observer might miss.

The message also doesn't second-guess the earlier backend selection. The assistant had already tested multiple backends and found flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM to be the working combination. It's not questioning that choice—it's looking for potentially faster alternatives. This shows a mature optimization mindset: first get something working, then look for improvements.

Broader Significance

In the arc of the session, message [msg 5942] is the moment where the assistant transitions from feature verification to performance optimization. The MTP question has been answered (yes, it's supported, and the weights are present), and now the focus shifts to squeezing out every last tok/s. This pivot is driven by the user's explicit concern about throughput ("Seems really low throughput"), which the assistant takes seriously.

The message also exemplifies the assistant's systematic approach to problem-solving. Rather than randomly trying backends, it reads the source code to understand the selection logic, traces configuration inheritance, and searches for architecture-specific guards. This reduces the number of trial-and-error experiments needed and builds a mental model of the system that will serve the rest of the session.

What follows this message (in later rounds) is a series of backend tests, the discovery that flashinfer_trtllm doesn't work on SM120, and the eventual deployment with flashinfer_cutlass and flashinfer_cudnn. But in this moment, the assistant is still in the investigative phase, gathering the information needed to make informed decisions. Message [msg 5942] is the hinge—the point where one investigation ends and another begins.