The Missing Config: Diagnosing MoE Kernel Performance on SM120 Blackwell

Introduction

In the long arc of deploying the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, few moments capture the essence of the debugging challenge more succinctly than message 660. This message, a single round in a multi-session conversation, contains two seemingly mundane bash tool calls: one listing pre-tuned MoE kernel configuration files, and another grepping server logs for MoE-related warnings. Yet within these outputs lies the critical insight that explains why the entire inference stack has been underperforming — and what must be done to fix it.

The Message

The assistant issued two parallel commands to the remote server:

ssh root@10.1.230.174 "ls /root/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/ | head -30"

And:

ssh root@10.1.230.174 "grep -i 'moe\|kernel config\|default.*config\|fused_moe' /root/sglang-server.log 2>/dev/null | head -20"

The results revealed a stark picture. The config directory contained pre-tuned kernel configurations for two GPU architectures: NVIDIA_B200.json (datacenter Blackwell, compute capability SM100) and NVIDIA_H100_80GB_HBM3.json (Hopper, SM90). For each architecture, configurations existed for expert counts of E=128 and various intermediate sizes (N=232, 464, 928, 1344, 1856, 2688). But there were zero configurations for SM120 — the architecture of the RTX PRO 6000 Blackwell GPUs actually in use.

The server log grep returned only two warnings: one about Transformers 5.2.0 compatibility, and another about forcing NSA prefill to use MLA on Blackwell. No MoE kernel config warnings appeared — meaning the server silently fell back to default (untuned) configurations without any notification.

Context and Motivation

This message did not occur in a vacuum. It was the direct result of a chain of observations and decisions spanning multiple conversation rounds. In the preceding message ([msg 657]), the user had noted: "still quite low power and lowish pcie bw, probably want to tune kernels." The assistant had just run benchmarks at 256 and 512 concurrency levels, achieving only ~880–912 total tok/s — far below the hardware's theoretical capacity. GPU power draw was hovering around 85W per GPU, a tiny fraction of the 600W TDP of each RTX PRO 6000 Blackwell card.

The assistant's reasoning, visible in [msg 658], was: "Good observation — low power (~83W idle) and the throughput plateau suggest we're not saturating the GPUs. MoE kernel tuning is the clear next step." It then queried the GLM-5 model configuration, discovering the Mixture-of-Experts parameters: E=256 routed experts, top_k=8 experts per token, moe_intermediate_size=2048, hidden_size=6144, and 78 transformer layers with the first 3 being dense.

With tensor parallelism 8 (TP8), each GPU handles 32 experts. The assistant then needed to understand what MoE kernel configurations were available, leading directly to message 660.

The Thinking Process Revealed

The assistant's choice of commands reveals a sophisticated diagnostic strategy. Rather than blindly running a tuning script, it first inspected what already existed. The ls command targeted the exact directory where sglang stores pre-tuned MoE kernel configurations — fused_moe_triton/configs/triton_3_5_1/ — corresponding to the installed Triton version. By listing only 30 entries, the assistant was sampling to see the naming pattern and architecture coverage.

The second command, grepping the server log for MoE-related messages, was equally deliberate. The assistant was checking whether the server had emitted any warnings about missing kernel configurations or falling back to defaults. The fact that no such warnings appeared is itself a significant finding: it means sglang silently uses default (untuned) kernels when no architecture-specific config exists, with no indication to the operator that performance is being left on the table.

Input Knowledge Required

To fully understand this message, one needs several pieces of context:

  1. The MoE architecture of GLM-5: The model uses 256 routed experts with top_k=8, meaning each token activates 8 experts. With TP8, each GPU handles 32 experts. The intermediate size (N) is 2048, but the config files show patterns like N=1344, N=1856, N=232 — these are different intermediate sizes for different model layers or configurations.
  2. The GPU architecture landscape: NVIDIA's Blackwell architecture comes in two major variants: SM100 (datacenter B200/B100) and SM120 (consumer/workstation RTX PRO 6000). These have different compute capabilities, shared memory sizes, and kernel support. Sglang's pre-tuned configs target SM100 (B200) and SM90 (H100), but not SM120.
  3. The Triton compiler versioning: Configs are stored per Triton version (3.1.0 through 3.5.1), because Triton's JIT compilation behavior and optimization heuristics change across versions. The configs are architecture-specific within each Triton version.
  4. The sglang MoE kernel pipeline: Sglang uses Triton-based fused MoE kernels that can be pre-tuned for specific (E, N, architecture) combinations. Without a matching config, the kernel falls back to default heuristics, which may be suboptimal for the specific hardware.

Output Knowledge Created

This message produced several critical pieces of knowledge:

  1. Confirmation of the missing SM120 configs: The most important finding — no pre-tuned MoE kernel configurations exist for the RTX PRO 6000 Blackwell (SM120) architecture. This is the root cause of the low GPU utilization and poor throughput.
  2. The config naming convention: Files follow the pattern E=<expert_count>,N=<intermediate_size>,device_name=<gpu_name>.json. This tells the assistant exactly how to name new configs generated through tuning.
  3. Silent fallback behavior: The server log contains no warnings about missing MoE configs. The system operates silently in a suboptimal state, which means operators must proactively check for this condition.
  4. The scope of tuning needed: With E=256 (per-GPU: 32) and N=2048, the assistant now knows the specific parameters for which tuning configs must be generated. The existing configs for E=128,N=1856 and E=128,N=2688 are close but not exact matches.

Assumptions and Potential Mistakes

The assistant made several assumptions in this message:

  1. That the config directory structure is the complete picture: The assistant assumed that all available pre-tuned configs are in this directory. This is reasonable but worth verifying — there might be additional configs elsewhere or generated at runtime.
  2. That missing configs explain the performance gap: While highly plausible, the assistant implicitly assumes that generating SM120-specific configs will close the performance gap. Other factors (PCIe topology, attention kernels, allreduce implementation) may also contribute.
  3. That the tuning script works for SM120: The assistant hasn't yet verified that the tuning scripts in benchmark/kernels/fused_moe_triton/ actually support SM120. There could be architecture-specific code paths that fail on the new hardware.
  4. That E=128 configs are relevant: The existing configs use E=128, but with TP8 each GPU handles E=32. The assistant may need to tune for E=32, not E=256. The config naming pattern includes the global expert count, so tuning for E=256 (or per-GPU E=32) requires understanding which parameter the tuning script expects.

The Broader Significance

This message represents a pivotal diagnostic moment in the deployment. Before it, the team knew performance was poor but didn't know why. After it, the path forward is clear: generate MoE kernel configurations for SM120 by running the tuning script, or find alternative MoE backends (like flashinfer_trtllm or CUTLASS) that support the architecture.

The missing SM120 configs also highlight a broader challenge in the AI infrastructure ecosystem: the rapid proliferation of GPU architectures (H100, B200, RTX PRO 6000, upcoming consumer Blackwell) outstrips the pace at which inference frameworks can generate and validate tuned kernels. Each new architecture requires not just driver support but a full tuning pass across all kernel configurations — a significant engineering effort that often lags behind hardware availability.

Conclusion

Message 660 is a masterclass in diagnostic efficiency. With two well-chosen commands, the assistant identified the root cause of a performance problem that had persisted across multiple deployment attempts. The absence of SM120 MoE kernel configurations in sglang's pre-tuned directory explains the low GPU utilization, the plateauing throughput, and the silent suboptimal operation. This finding directly informs the next steps: either running the tuning script to generate SM120-specific configs, or exploring alternative MoE backends. In either case, the message demonstrates that sometimes the most valuable output is not a solution, but a precise diagnosis that transforms an amorphous performance problem into a well-defined engineering task.