Reading the Map: How One `grep` Command Revealed the Landscape of MoE Backends in SGLang

The Message

In the midst of an intensive optimization session targeting the GLM-5-NVFP4 model on eight RTX PRO 6000 Blackwell GPUs, the assistant issued a single, deceptively simple command:

ssh root@10.1.230.174 'grep -A20 "class MoeRunnerBackend" /root/sglang/python/sglang/srt/layers/moe/utils.py 2>/dev/null'

The output returned the definition of an enumeration:

class MoeRunnerBackend(Enum):

    AUTO = "auto"
    DEEP_GEMM = "deep_gemm"
    TRITON = "triton"
    TRITON_KERNELS = "triton_kernel"
    FLASHINFER_TRTLLM = "flashinfer_trtllm"
    FLASHINFER_CUTLASS = "flashinfer_cutlass"
    FLASHINFER_MXFP4 = "flashinfer_mxfp4"
    FLASHINFER_CUTEDSL = "flashinfer_cutedsl"
    CUTLASS = "cutlass"
    MARLIN = "marlin"

This is the subject message — message index 1232 in a long conversation spanning dozens of rounds, multiple subagents, kernel upgrades, and countless benchmarks. On its surface, it is nothing more than a source-code lookup. But within the arc of this optimization journey, it represents a critical turning point: the moment when the assistant, having exhausted several higher-level optimization strategies, turned to the deepest layer of the inference stack — the actual kernel backend that computes the Mixture-of-Experts (MoE) matrix multiplications.

The Context: A 30x Performance Gap

To understand why this simple grep matters, one must understand the crisis that preceded it. The assistant had spent the entire session (spanning segments 5 through 10) trying to squeeze performance out of the GLM-5-NVFP4 model — a massive sparse MoE language model with FP4 quantization, running on cutting-edge Blackwell GPUs with the SM120 architecture.

The numbers told a painful story. The theoretical maximum single-stream performance, computed with meticulous care, was approximately 309 tokens per second — corresponding to a decode latency of about 3.2 milliseconds per token. This theoretical limit was derived from the GPU's memory bandwidth (1,800 GB/s) and the model's weight footprint per GPU (2.86 GB), assuming perfect streaming of weights through the compute units.

Reality was far less kind. Actual measured decode latency was approximately 97 milliseconds per token — a staggering 30x gap. The model was achieving only about 3.4% of its theoretical peak efficiency. Every optimization avenue the assistant had explored had yielded incremental gains at best. Piecewise CUDA graphs were blocked by architectural incompatibilities. MSCCLPP (a collective communication library) showed minimal improvement. Expert parallelism (EP8) crashed under load with illegal memory access errors. Even after a major kernel upgrade to Linux 6.14.11 and a comprehensive system audit that fixed CPU governor settings, NUMA balancing, PCIe read request sizes, and deep C-states, the fundamental problem remained.

A diagnostic tool built in the same segment had measured the latency of individual decode components. The results were illuminating: simulated BF16 GEMMs and AllReduce operations accounted for only about 8.9 milliseconds of the 95-millisecond decode time. The remaining ~86 milliseconds were consumed by something else — and the evidence pointed squarely at the FP4 GEMM kernels, the MoE routing logic, and the attention mechanism.

Why This Message Was Written

The assistant's reasoning, visible in the messages immediately preceding this one ([msg 1229], [msg 1230], [msg 1231]), reveals a deliberate pivot. After noting that "rather than spending time on profiling (which would require modifying the forward pass)," the assistant decided to "focus on the flashinfer_cutedsl backend which could fundamentally change the GEMM performance."

This decision reflects a key insight: the FP4 GEMM kernels — the actual GPU kernel code that performs the tiny matrix multiplications (M=1, N=256, K=6144) for each expert in the MoE layer — were the primary bottleneck. If these kernels could be made more efficient, the entire decode pipeline would accelerate. The flashinfer_cutedsl backend represented a potential path to better kernel performance because CuteDSL (a domain-specific language built on NVIDIA's CUTLASS library) could potentially generate more efficient CUDA kernels for the specific GEMM shapes and data types used by the FP4-quantized model.

But before the assistant could try this backend, they needed to understand the landscape. Three reconnaissance steps were taken:

[msg 1230]: A search for "cutedsl" references in the sglang source code, which confirmed that enable_flashinfer_cutedsl_moe was a supported feature in the quantization code.
[msg 1231]: A search in server_args.py that confirmed "flashinfer_cutedsl" was listed as a valid backend name in the command-line argument parser.
[msg 1232] (the subject message): The definitive lookup — reading the actual MoeRunnerBackend enum class to see all available backends and understand the full set of options. This third step was the most important. It wasn't enough to know that flashinfer_cutedsl existed; the assistant needed to see the complete menu of choices to understand where flashinfer_cutedsl fit in the ecosystem, what other backends were available, and how they related to each other.

The Landscape Revealed

The MoeRunnerBackend enum revealed ten distinct backends:

AUTO: Automatic selection, delegating the choice to the system.
DEEP_GEMM: Likely refers to DeepGEMM, a library for efficient GEMM operations on deep learning workloads.
TRITON and TRITON_KERNELS: Backends using OpenAI's Triton language for custom kernel generation.
FLASHINFER_TRTLLM: Integration with NVIDIA's TensorRT-LLM inference framework via FlashInfer.
FLASHINFER_CUTLASS: Using NVIDIA's CUTLASS template library through FlashInfer.
FLASHINFER_MXFP4: A specialized backend for FP4 (micro-scaling FP4) quantization — directly relevant to GLM-5-NVFP4.
FLASHINFER_CUTEDSL: Using NVIDIA's CuteDSL (a DSL built on CUTLASS) through FlashInfer.
CUTLASS: Direct CUTLASS backend without FlashInfer mediation.
MARLIN: A backend optimized for 4-bit quantized models (Marlin sparse kernels). The output was truncated after the first two methods (is_auto and is_deep_gemm), but the critical information was captured: the full set of backend names, confirming that flashinfer_cutedsl was indeed a valid, first-class option.

The Thinking Process

The assistant's thinking, visible in the surrounding messages, reveals a methodical, hypothesis-driven approach to optimization. The chain of reasoning went as follows:

Problem identification: Single-stream decode latency is ~97ms vs ~3.2ms theoretical. Something is consuming ~86ms beyond the expected compute and communication costs.
Diagnosis: The diagnostic tool showed that BF16 GEMMs and AllReduce only account for ~8.9ms. The remaining time must be in FP4 GEMM kernels, MoE routing, or attention.
Hypothesis: The FP4 GEMM kernels are inefficient for the specific matrix shapes (small, narrow matrices with M=1) on the Blackwell SM120 architecture. Different kernel backends may have dramatically different performance characteristics.
Action: Explore alternative MoE backends, starting with flashinfer_cutedsl which may generate more efficient CUDA kernels for FP4 data types.
Reconnaissance: Search the codebase to confirm the backend exists, how to enable it, and what alternatives are available. This is not random exploration — it is a structured investigation following the scientific method. Each step builds on the previous one, and each finding narrows the search space.

Assumptions and Knowledge Required

This message, and the reasoning behind it, relies on several layers of domain knowledge:

Understanding of MoE architecture: The reader must know that Mixture-of-Experts layers use multiple "expert" sub-networks, with a routing mechanism that selects which experts to activate for each input token. The GEMM operations in MoE layers are unusual — they involve small, narrow matrices (batch size 1, narrow hidden dimensions) that are notoriously difficult to accelerate on GPUs.
Knowledge of GPU kernel backends: The distinction between CUTLASS, CuteDSL, Triton, TensorRT-LLM, and Marlin is meaningful. Each represents a different approach to generating CUDA kernels, with different trade-offs in terms of flexibility, performance, and hardware support.
Familiarity with the Blackwell architecture: The SM120 compute architecture on Blackwell GPUs has specific characteristics (e.g., FP4 tensor core support, register file size, shared memory capacity) that affect kernel performance.
Understanding of sglang's architecture: The assistant knows that the MoE backend is configured via a command-line argument (--moe-a2a-backend or similar) and that changing it requires restarting the server. The key assumption underlying this message is that the MoE backend choice can materially affect FP4 GEMM kernel performance. This is a reasonable assumption — different kernel generation systems produce different CUDA code, and for the specific matrix shapes and data types used by GLM-5-NVFP4, some backends may be significantly more efficient than others. However, it is also possible that all backends converge to similar performance for these particular GEMM shapes, or that the bottleneck lies elsewhere (e.g., in the attention mechanism or the routing logic).

Output Knowledge Created

This message produced concrete, actionable knowledge:

Confirmation of backend availability: flashinfer_cutedsl is a valid, supported backend in the sglang codebase, not a hypothetical or experimental feature.
Complete backend inventory: The full list of ten backends, providing a map of the optimization landscape. The assistant now knows what options exist and can reason about which ones to try.
Code location: The MoeRunnerBackend enum is defined in /root/sglang/python/sglang/srt/layers/moe/utils.py, giving the assistant a precise location for further investigation (e.g., checking how each backend is initialized, what dependencies they require, and what hardware they support).
Validation of the exploration path: Finding flashinfer_cutedsl as a named member of the enum validates the assistant's hypothesis that this is a real, supported option worth trying.

The Broader Significance

This message exemplifies a pattern that recurs throughout the entire optimization session: the assistant repeatedly cycles between measurement, hypothesis formation, and codebase exploration. Each cycle narrows the focus. The theoretical analysis identified the magnitude of the gap. The diagnostic tool identified the FP4 GEMM kernels as the likely culprit. The codebase exploration identified flashinfer_cutedsl as a potential remedy.

What makes this message interesting is not the command itself — it is a trivial grep — but the decision-making context that made it necessary. The assistant could have blindly tried random backends, restarting the server each time and measuring the result. Instead, it invested time in understanding the landscape first: reading the source code, confirming the backend names, and building a mental model of the available options. This is the hallmark of a systematic optimization approach — measure, hypothesize, investigate, then act.

The message also reveals something about the nature of AI-assisted coding sessions. The assistant is not just executing commands; it is reasoning about trade-offs, forming hypotheses, and making strategic decisions about where to invest effort. The grep command is a tool of thought — a way of reading the source code as a map, understanding the terrain before committing to a path.

What Came Next

After this message, the assistant would go on to try the flashinfer_cutedsl backend, along with other backends, in a systematic attempt to close the 30x performance gap. The outcome of those experiments would be documented in subsequent messages, building on the foundation laid by this reconnaissance. Whether flashinfer_cutedsl proved to be the silver bullet or just another incremental improvement, the decision to explore it was grounded in a thorough understanding of the codebase — an understanding that began with a single grep command.

Reading the Map: How One grep Command Revealed the Landscape of MoE Backends in SGLang