The Last MoE Backend: Systematic Kernel Exploration on Blackwell GPUs

The Message

[bash] ssh -f 10.1.230.175 'source ~/ml-env/bin/activate && export NCCL_IB_DISABLE=1 && export NCCL_P2P_LEVEL=PHB && export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 && export NCCL_MIN_NCHANNELS=8 && export OMP_NUM_THREADS=8 && export SAFETENSORS_FAST_GPU=1 && export CUDA_HOME=/usr/local/cuda-12.8 && nohup python3 -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 --reasoning-parser glm45 --tool-call-parser glm47 --trust-remote-code --tp 8 --mem-fraction-static 0.92 --max-running-requests 64 --kv-cache-dtype auto --quantization modelopt_fp4 --attention-backend flashinfer --fp8-gemm-backend cutlass --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutedsl --disable-custom-all-reduce --enable-flashinfer-allreduce-fusion --host 0.0.0.0 --port 8000 > ~/sglang-glm5.log 2>&1 &'

At first glance, this message appears to be a routine server restart — just another SGLang launch command with a slightly different flag. But this single bash invocation represents the culmination of a deep, methodical investigation into the MoE kernel landscape on NVIDIA's newest Blackwell architecture. It is the last alternative in a systematic search, the final variable in an exhaustive configuration sweep, and the moment before the assistant pivots from exploration to conclusion.

Context: The Quest for a Working MoE Backend

To understand why this message was written, we must trace the investigation that preceded it. The assistant had been deploying the GLM-5-NVFP4 model — a massive Mixture-of-Experts (MoE) language model with 256 experts — across 8 RTX PRO 6000 Blackwell GPUs (SM120 architecture). The journey had already been arduous: the initial deployment crashed with NaN errors during decode, which was resolved by selecting specific NSA attention backends (trtllm for both decode and prefill). With a working baseline established at roughly 225 output tokens per second under batch load, the assistant turned its attention to performance optimization.

The core question became: which MoE runner backend delivers the best throughput on these brand-new SM120 GPUs? SGLang supports multiple backends through FlashInfer, each implementing the fused MoE computation differently:

The Reasoning Behind the Launch

Message 298 is the direct execution of a decision made in message 297. In that preceding reasoning block, the assistant had just completed an investigation into FlashInfer's autotuning infrastructure. It discovered that FlashInfer maintains per-device tuning configuration files mapping kernel configurations to optimal tile sizes, and that configuration files existed for NVIDIA B200 and GB200 (both SM100 datacenter Blackwell variants) but not for the RTX PRO 6000 Blackwell Server Edition (SM120). When no config file exists, the autotuner falls back to default configuration 0 with tile config -1 — a suboptimal setting.

More importantly, the assistant realized that the autotuner only applies to the flashinfer_trtllm path, which uses TRT-LLM-style kernels with multiple tactic choices. The flashinfer_cutlass path uses a fixed CUTLASS kernel where the only tunable parameter is tune_max_num_tokens. This meant that the investigation into autotuning was, in a sense, a dead end for the current configuration.

With this understanding, the assistant reframed the problem. The performance bottleneck was not kernel tuning per se but the sheer number of small kernel launches required by the model architecture: 78 layers, each with multiple kernels per layer, routing through 256 experts. This many small operations on a PCIe-interconnected GPU cluster creates a latency-bound scenario where kernel launch overhead and cross-GPU communication dominate.

The assistant's final assessment was clear: "Let me quickly test flashinfer_cutedsl (the CuteDSL path) as the last alternative, then record final results." Message 298 is that test.

How Decisions Were Made

The decision to try flashinfer_cutedsl was not arbitrary — it was the logical conclusion of a systematic elimination process. The assistant had established a clear decision tree:

  1. Start with flashinfer_cutlass (the default, most widely tested backend). ✓ Tested, worked.
  2. Try flashinfer_trtllm (potentially faster TRT-LLM kernels). ✗ Crashed, SM100-only.
  3. Try flashinfer_cutedsl (CuteDSL grouped GEMM, different kernel implementation). → This message. The decision was also informed by the earlier discovery that FlashInfer had SM120-specific source files: fp4_gemm_cutlass_sm120.cu, group_gemm_sm120_binding.cu, and fp4_gemm_template_sm120.h. The presence of these files confirmed that the FlashInfer developers had invested in SM120 support, making it plausible that the CuteDSL path might have received similar attention. The specific command-line parameters in message 298 reflect a carefully curated set of options accumulated over hours of debugging. Each flag tells a story: - --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm: The critical fix that resolved the NaN crash during decode. These were discovered through iterative experimentation with attention backends. - --mem-fraction-static 0.92: Increased from the default to maximize GPU memory utilization for KV cache, tested successfully in a previous run. - --disable-custom-all-reduce and --enable-flashinfer-allreduce-fusion: Network configuration flags to work around the PCIe-only interconnect (NCCL P2P level set to PHB, meaning "Peer-to-Peer over Host Bridge" — the GPUs communicate through the host memory, not directly via NVLink). - --tp 8: Tensor parallelism across all 8 GPUs. - --quantization modelopt_fp4: The NVFP4 quantization format used by the GLM-5-NVFP4 model. The environment variables also encode critical infrastructure knowledge: NCCL_IB_DISABLE=1 disables InfiniBand (not present), NCCL_P2P_LEVEL=PHB forces PCIe peer-to-peer through host bridges, and CUDA_HOME=/usr/local/cuda-12.8 points to a secondary CUDA toolkit installation (the system also has CUDA 13.1, but FlashInfer and other components were built against 12.8).

Assumptions and Their Validity

The message carries several implicit assumptions:

Assumption 1: The flashinfer_cutedsl backend will at least load and run without crashing. This was not guaranteed. The flashinfer_trtllm backend had already crashed on SM120, and the CuteDSL path could have similar architecture-specific code paths. The assistant was operating without documentation — this was exploratory testing on a brand-new GPU architecture.

Assumption 2: The CuteDSL path might offer different performance characteristics. The assistant hypothesized that a grouped GEMM implementation (CuteDSL) might handle the small-batch, many-expert routing pattern differently than the CUTLASS fused kernel. This was a reasonable hypothesis given that the two implementations use fundamentally different code generation approaches.

Assumption 3: The server configuration (memory fraction, CUDA graphs, attention backends) would remain valid with a different MoE backend. The assistant kept all other parameters identical, assuming they were orthogonal to the MoE runner choice. This was a reasonable experimental control — changing only one variable at a time.

Assumption 4: The remote server was in a clean state. The assistant had just killed the previous SGLang process (pkill -9 -f sglang; sleep 3; echo done in message 297), but there was no explicit verification that GPU memory was freed, NCCL resources were released, or that no zombie processes remained. The sleep 3 was a minimal wait.

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains:

SGLang server architecture: Understanding what --moe-runner-backend controls — the kernel implementation used for the Mixture-of-Experts forward pass, which is the computational heart of MoE models like GLM-5-NVFP4.

FlashInfer kernel taxonomy: Knowing that flashinfer_cutlass, flashinfer_trtllm, and flashinfer_cutedsl are three distinct kernel implementations within the FlashInfer library, each with different code generation paths, CUDA architecture requirements, and performance characteristics.

Blackwell GPU architecture: Understanding the SM120 vs. SM100 distinction — the RTX PRO 6000 uses SM120 (workstation Blackwell) while datacenter parts like B200 and GB200 use SM100. This distinction was critical because the flashinfer_trtllm backend was compiled for SM100 and failed on SM120.

NCCL and multi-GPU communication: The environment variables controlling NCCL behavior (NCCL_P2P_LEVEL=PHB, NCCL_IB_DISABLE=1) reflect an understanding of the GPU interconnect topology — or lack thereof. These GPUs are connected via PCIe only, without NVLink, and are further virtualized through Proxmox/KVM, which adds additional latency.

Model quantization: The modelopt_fp4 quantization format and the NVFP4 block scaling scheme used by the GLM-5-NVFP4 model. The MoE backends must support this specific quantization format.

Output Knowledge Created

This message, combined with its result (which would arrive in the next round), would create several pieces of knowledge:

  1. Whether flashinfer_cutedsl works on SM120: A binary answer — does it load, initialize, and produce coherent output without crashing?
  2. Comparative performance data: How does the CuteDSL path compare to the CUTLASS baseline in terms of throughput (tokens/second), latency, GPU utilization, and power draw?
  3. The completeness of the backend exploration: If CuteDSL also fails or performs worse, the assistant would have exhausted the available MoE backends and could conclude that the CUTLASS baseline represents the best achievable performance with current software.
  4. Diagnostic signal for future work: If CuteDSL performs differently, it provides clues about where the bottleneck lies — in the GEMM computation itself, in the routing logic, or in the cross-GPU communication.

The Thinking Process

The reasoning visible in the surrounding messages reveals a methodical, hypothesis-driven approach. The assistant treated the MoE backend selection as an experimental variable to be systematically tested, controlling for all other parameters. Each test was preceded by a reasoning step that synthesized the results of the previous test and formulated the next hypothesis.

The investigation into FlashInfer's autotuner (messages 282–297) is particularly revealing of the assistant's thinking style. Rather than accepting the performance as-is, the assistant dug into the source code, examining autotuner.py, checking for device-specific config files, tracing the load_from_file fallback logic, and ultimately determining that the autotuner didn't apply to the CUTLASS path at all. This deep-dive was motivated by a specific observation: the GPUs were at 100% utilization but only 55% power, suggesting untapped performance potential. The assistant wanted to know: is there a tuning knob we haven't turned?

The conclusion that "the CUTLASS SM120 kernels are what they are" represents a moment of intellectual closure — accepting that the current kernel implementation is fixed and the path to better performance lies elsewhere (perhaps in reducing kernel launch overhead, increasing batch size, or addressing the PCIe communication bottleneck).

Message 298 is the final experiment in this line of inquiry. After it, the assistant planned to "record final results" and move on to higher-level performance considerations like expert parallelism and virtualization overhead. This makes the message a pivot point — the last variable in the kernel exploration before the assistant zooms out to the system-level bottlenecks.

Broader Significance

This message exemplifies a common pattern in ML infrastructure engineering: the systematic exploration of a configuration space where documentation is sparse, the hardware is new, and the software stack is evolving. The assistant had no manual telling it which MoE backend to use on SM120 GPUs — it had to discover this through experimentation. The three-backend taxonomy (cutlass, trtllm, cutedsl) represents different engineering approaches to the same mathematical problem, and their behavior on new hardware is an empirical question.

The message also highlights the layered complexity of modern ML serving stacks. A single launch command encodes decisions about attention algorithms, quantization formats, communication protocols, memory management, and kernel implementations — each with its own failure modes, compatibility constraints, and performance characteristics. The assistant's ability to navigate this complexity, isolate variables, and draw conclusions from ambiguous signals is a form of expertise that this single message only hints at.

In the end, whether flashinfer_cutedsl succeeds or fails, message 298 represents the disciplined execution of a well-formed experimental plan — and that is the mark of rigorous engineering practice.