Probing the MoE Runner Dispatch: A Systematic Search for the Right Blackwell GEMM Backend

In the high-stakes world of deploying large language models on cutting-edge hardware, every microsecond counts. Message [msg 5949] captures a pivotal moment in an intensive optimization session: the assistant, having just confirmed that one GEMM backend (flashinfer_trtllm) supports the Blackwell SM120 architecture for Mixture-of-Experts (MoE) computation, pivots to systematically enumerate all available backend options. This brief but consequential message — consisting of a single reasoning statement and one grep command — embodies the methodical, data-driven approach that characterizes the entire session.

The Message

The assistant writes:

The flashinfer_trtllm MoE runner imports fp4_quantize from flashinfer on SM120. So it does support SM120. But there's also flashinfer_cutlass and flashinfer_cutedsl for MoE. Let me check the MoE runner dispatch:

>

``bash ssh root@[REDACTED_IP] 'grep -n "flashinfer_cutlass\|flashinfer_cutedsl\|cutlass.*moe\|cutedsl" /root/sglang-main/python/sglang/srt/layers/moe/moe_runner/runner.py 2>&1 | head -20' ``

On its surface, this is a straightforward probe. But beneath it lies a complex web of reasoning about GPU architecture compatibility, backend dispatch logic, and the trade-offs between performance and correctness on a novel hardware platform.

Context: The Blackwell Backend Puzzle

To understand why this message matters, we must step back into the broader context of the session. The team is deploying the Qwen3.5-397B-A17B-NVFP4 model — a 397-billion-parameter Mixture-of-Experts model quantized to NVFP4 (NVIDIA's 4-bit floating-point format) — on a machine equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs. These GPUs have compute capability 12.0 (SM120), which places them in a peculiar position: they are Blackwell-generation hardware, but they are not the same as the B200 or GB200 GPUs (compute capability 10.0, SM100) that most SGLang optimization work has targeted.

This architectural distinction creates a fundamental tension. Many GEMM (General Matrix Multiply) backends in SGLang were developed and tuned for SM100 Blackwell GPUs. The SM120 variant — used in the RTX PRO 6000 — has a different number of Streaming Multiprocessors (SMs), different cache hierarchies, and potentially different instruction scheduling characteristics. A backend that works flawlessly on a B200 might crash, produce NaN values, or deliver suboptimal throughput on an SM120 GPU.

In the messages immediately preceding [msg 5949], the assistant had been systematically investigating this compatibility landscape. It had discovered that the flashinfer_trtllm MoE runner file (at /root/sglang-main/python/sglang/srt/layers/moe/moe_runner/flashinfer_trtllm.py) imports is_sm120_supported and fp4_quantize from flashinfer, suggesting that this backend has explicit support for SM120. This was an encouraging finding — one backend was confirmed to work.

The Reasoning: Why This Message Was Written

Message [msg 5949] represents the natural next step in a systematic exploration. The assistant's reasoning can be reconstructed as follows:

  1. Premise: We have confirmed that flashinfer_trtllm supports SM120 for MoE computation.
  2. Observation: The MoE runner directory contains multiple backend implementations — not just flashinfer_trtllm, but also flashinfer_cutlass and flashinfer_cutedsl (as seen in the directory listing from [msg 5947]).
  3. Question: Are these other backends also available for SM120? And if so, how does the dispatch logic choose between them?
  4. Action: Examine the dispatch file (runner.py) to see how backends are selected and whether SM120-specific conditions exist. This is classic systematic debugging: having confirmed one path works, enumerate all alternatives to understand the full landscape before making a decision. The assistant is not satisfied with a single working backend — it wants to know the complete menu of options.

Assumptions Embedded in the Probe

The message carries several implicit assumptions:

Assumption 1: The dispatch logic is centralized. By grepping runner.py for backend names, the assistant assumes that the backend selection logic lives in a single dispatch file. This is a reasonable assumption — SGLang's codebase is well-structured — but it's not guaranteed. Some backend selection might happen in configuration files, environment variables, or even at the C++/CUDA level.

Assumption 2: Backend names in the code correspond to actual functional backends. The assistant assumes that flashinfer_cutlass and flashinfer_cutedsl are real, usable backends, not deprecated or placeholder entries. This assumption is supported by the earlier research task ([msg 5930]) which identified these as valid options in the Fp4GemmRunnerBackend enum and the MoeRunnerBackend enum.

Assumption 3: SM120 support is backend-specific. The assistant treats each backend as potentially having different SM120 compatibility. This is correct — different backends use different CUDA kernel implementations (TensorRT-LLM, CUTLASS, Cute-DSL, cuDNN), each with its own compilation targets and architecture checks.

Assumption 4: The grep pattern will capture all relevant dispatch logic. The pattern "flashinfer_cutlass\|flashinfer_cutedsl\|cutlass.*moe\|cutedsl" is designed to catch both exact backend names and any related conditional logic. However, it might miss dispatch logic that uses enum values, string variables, or indirect references.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Knowledge of SGLang's MoE architecture: That MoE computation is handled by a "runner" abstraction with multiple backend implementations, each optimized for different hardware.
  2. Understanding of Blackwell GPU variants: That SM100 (B200/GB200) and SM120 (RTX PRO 6000) are different compute capabilities within the Blackwell family, requiring separate kernel compilation targets.
  3. Familiarity with GEMM backends: That flashinfer_trtllm, flashinfer_cutlass, and flashinfer_cutedsl are different kernel implementations for FP4 matrix multiplication, each with different performance characteristics and hardware requirements.
  4. Context from preceding messages: That the assistant has been investigating throughput issues, discovered MTP support in the model, and is now systematically testing backends to find the fastest configuration for SM120.

Output Knowledge Created

This message, combined with the response it triggers, creates several pieces of knowledge:

  1. Backend availability on SM120: The grep results will reveal which backends are registered in the dispatch logic and whether any SM120-specific conditions exist.
  2. Dispatch architecture: Understanding how SGLang selects between MoE runners — whether it's automatic, configurable via flags, or conditional on hardware detection.
  3. Enumeration of viable paths: A complete list of MoE runner backends that could potentially work on SM120, forming the basis for subsequent benchmarking. This knowledge is immediately actionable. The assistant will use it to design the next round of experiments — testing each backend for correctness (does it produce valid output?) and performance (what throughput does it achieve?).

The Thinking Process

The message reveals a clear thinking pattern. The assistant is operating in a "confirm and expand" mode:

  1. Confirm: Verify that flashinfer_trtllm supports SM120 (done in [msg 5948] by reading the source file).
  2. Announce: State the confirmed finding explicitly ("So it does support SM120").
  3. Identify gaps: Recognize that other backends exist and need similar verification ("But there's also flashinfer_cutlass and flashinfer_cutedsl for MoE").
  4. Probe: Execute a targeted command to fill the gap. The use of "But there's also..." is telling — it signals a shift from confirmation to exploration. The assistant is not concluding its investigation; it's expanding the search space. This is characteristic of thorough debugging: never assume that the first working option is the best option.

What Happens Next

The grep command in this message will reveal the dispatch logic in runner.py. In the subsequent messages ([msg 5950] onward), the assistant will use this information to test multiple backend configurations, eventually discovering that flashinfer_cutlass works correctly for MoE on SM120 while flashinfer_trtllm and flashinfer_cutedsl crash or produce garbage output. This systematic enumeration — starting with the confirmation in [msg 5949] — is what ultimately leads to a stable, high-performance deployment configuration.

Conclusion

Message [msg 5949] is a small but critical node in a larger investigative network. It demonstrates the assistant's methodical approach to hardware compatibility: confirm one path, enumerate alternatives, probe systematically, and let the data guide the decision. In the high-dimensional optimization space of LLM inference on novel hardware, such systematic reasoning is not a luxury — it is the only reliable path to a working, performant deployment.