The Pivot Point: MoE Kernel Tuning Begins for GLM-5 on SM120 Blackwell

Introduction

In the long and winding journey of deploying the 744-billion-parameter GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment where the trajectory of the entire effort shifts. Message 659 in this opencode session is that moment. It is a short, unassuming message — three bash commands, a brief summary of model parameters, and no dramatic declarations — yet it represents a critical pivot from broad performance exploration to targeted, deep kernel-level optimization. Understanding why this message was written, what knowledge it synthesizes, and what assumptions it carries is essential to grasping the full arc of this deployment story.

The Context: A Performance Plateau

To appreciate message 659, one must understand what came before it. The assistant had already accomplished a remarkable amount: it had set up an LXC container on a Proxmox host to bypass VFIO/IOMMU overhead, resolved a CUDA initialization failure caused by NVIDIA's HMM feature, confirmed bare-metal GPU topology with P2P access at 53 GB/s within NUMA nodes, and launched a working SGLang inference server for the GLM-5-NVFP4 model. Yet despite all this infrastructure work, the performance numbers were stubbornly plateauing. At 128 concurrent requests, the server achieved approximately 806 total tokens per second — respectable, but far below the 1,000+ tok/s target and dramatically below the 5,800+ tok/s that prior research had achieved with a different model (Kimi K2) on the same hardware.

The user's observation at message 657 crystallized the frustration: "still quite low power and lowish pcie bw, probably want to tune kernels." The GPUs were drawing around 83 watts each at idle — a tiny fraction of their 600-watt thermal design power. Something was fundamentally preventing the hardware from being utilized. The assistant's response at message 658 acknowledged this and began checking the GLM-5 model's MoE (Mixture-of-Experts) parameters, setting the stage for the decisive action in message 659.

What Message 659 Actually Says

The message opens with a concise synthesis:

So GLM-5 has E=256, N=2048 (moe_intermediate_size), top_k=8. With TP8, each GPU handles E=256/8=32 experts with N=2048. Let me check what the tuning script expects and what dtype we need for NVFP4.

This single sentence packs an enormous amount of reasoning. The assistant has just retrieved the model configuration from the Hugging Face snapshot at /shared/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/config.json (in the previous message). From that config, it extracted n_routed_experts: 256, moe_intermediate_size: 2048, and num_experts_per_tok: 8. It then performed a mental calculation: with tensor parallelism of 8 (TP8), each GPU is responsible for 256 ÷ 8 = 32 experts, each with an intermediate size of 2048. This decomposition is crucial because the MoE kernel tuning scripts require these exact dimensions (E and N) to generate optimal kernel configurations.

The assistant then dispatches three bash commands in parallel, each probing a different aspect of the MoE tuning infrastructure:

  1. Listing the tuning scripts: ls /root/sglang/benchmark/kernels/fused_moe_triton/ — This reveals the available tuning tools, including tuning_fused_moe_triton.py and tuning_fused_moe_triton_sep.py, which are the scripts used to generate hardware-specific MoE kernel configurations.
  2. Listing existing kernel configs: ls /root/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/ — This shows the directories of pre-tuned kernel configurations organized by Triton version (3.1.0 through 3.5.1). The assistant is checking whether any SM120-specific configs already exist for this model architecture.
  3. Searching the server log: grep -r 'Using default MoE' /root/sglang-server.log — This is a diagnostic probe to see if the running server is using default (suboptimal) MoE kernel configurations, which would confirm the hypothesis that kernel tuning is needed.

The Reasoning and Motivation

Message 659 is written because the assistant has reached a fork in the road. The higher-concurrency benchmarks run in messages 655 and 656 showed that throughput was scaling only modestly from 256 concurrency (879.87 tok/s) to 512 concurrency (912.70 tok/s). The curve was flattening, suggesting that the bottleneck was not request throughput but rather per-token computation efficiency. The GPUs were not being saturated — they were waiting on something.

The assistant's reasoning, visible in the trajectory of the conversation, follows a clear chain:

  1. Observation: Throughput plateaus as concurrency increases, GPU power stays near idle.
  2. Hypothesis: The MoE kernel configurations are suboptimal for SM120 (consumer Blackwell architecture). The RTX PRO 6000 has SM120 compute capability, while most pre-tuned configs target H100 (SM90) or B200 (SM100). Without SM120-specific configs, the Triton compiler either falls back to generic kernels or uses configs designed for a different architecture, leading to poor occupancy and memory utilization.
  3. Action: Verify the model's MoE dimensions, locate the tuning infrastructure, and prepare to generate custom kernel configs. This reasoning is grounded in prior research documented in the local FINDINGS.md file, which the assistant has been consulting throughout the session. That file contains detailed notes from deploying Kimi K2 (another large MoE model) on the same RTX PRO 6000 hardware, where MoE kernel tuning was identified as the single biggest performance lever.

Assumptions Made

The assistant makes several assumptions in this message, some explicit and some implicit:

Explicit assumption: The MoE tuning scripts at /root/sglang/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py are the correct tools for generating SM120-specific kernel configs for GLM-5. This assumption is reasonable given the prior research, but it carries risk: the tuning scripts were designed for Triton-based MoE kernels, and the server is currently using --moe-runner-backend flashinfer_cutlass, which may use a different kernel path (CUTLASS-based rather than Triton-based). The assistant does not yet know whether tuning Triton configs will benefit the flashinfer_cutlass backend.

Implicit assumption: The model dimensions (E=256, N=2048) are the only parameters needed for tuning. In reality, the tuning scripts also need information about the GPU's shared memory capacity, register file size, and memory bandwidth characteristics — but these are typically detected automatically from the GPU architecture.

Implicit assumption: The server log search for "Using default MoE" will reveal whether suboptimal configs are in use. This assumes that SGLang logs a warning when falling back to default configs. If the logging is absent or uses different phrasing, the search will miss it.

Implicit assumption: The NVFP4 quantization format (modelopt_fp4) does not fundamentally change the MoE kernel tuning problem — that is, the same kernel configs that work for FP8/BF16 MoE will also work for NVFP4. This is a significant assumption, as NVFP4 uses 4-bit floating point with non-standard scale factors, which may require different tile sizes or memory access patterns.

Input Knowledge Required

To fully understand message 659, a reader needs knowledge in several domains:

  1. Mixture-of-Experts architecture: Understanding that E=256 means 256 expert networks, N=2048 is the intermediate dimension of each expert's feed-forward network, and top_k=8 means 8 experts are activated per token. Also understanding that tensor parallelism (TP8) shards the experts across GPUs, so each GPU handles 32 experts.
  2. SGLang's MoE kernel infrastructure: Knowing that SGLang stores pre-tuned kernel configurations in versioned directories under fused_moe_triton/configs/, and that the tuning scripts generate JSON configs mapping model dimensions to optimal kernel launch parameters (block sizes, tile sizes, number of stages).
  3. SM120 vs SM100 differences: Understanding that the RTX PRO 6000 uses SM120 (consumer Blackwell) with 100KB shared memory per SM, while datacenter Blackwell (B200, SM100) has 228KB. This difference means Triton kernels tuned for SM100 may fail or perform poorly on SM120 due to shared memory exhaustion.
  4. NVFP4 quantization: Knowing that NVFP4 is a 4-bit floating point format used by NVIDIA's ModelOpt toolkit, and that it requires specific kernel support in the MoE computation path.
  5. The broader deployment context: Understanding that this is an LXC container on a Proxmox host, that the server is already running with specific flags (--moe-runner-backend flashinfer_cutlass, --attention-backend flashinfer, etc.), and that prior benchmarks have established a baseline around 800-900 tok/s.

Output Knowledge Created

Message 659 creates several pieces of actionable knowledge:

  1. Confirmation of tuning infrastructure availability: The tuning scripts exist and are accessible at the expected path. The config directories show that SGLang supports multiple Triton versions (3.1.0 through 3.5.1), which means the assistant can generate configs compatible with the installed Triton 3.5.1.
  2. Absence of SM120-specific configs: By listing the config directories and seeing only version-numbered directories (not GPU-specific ones), the assistant implicitly confirms that no pre-tuned SM120 configs exist for GLM-5's dimensions. This is reinforced by the server log search, which (as we see in subsequent messages) does not find the expected "Using default MoE" warning — suggesting either that the flashinfer_cutlass backend doesn't use Triton MoE configs at all, or that the logging is absent.
  3. Model dimension confirmation: The assistant has now confirmed E=256, N=2048, top_k=8, and computed the per-GPU expert count (32). These numbers will be passed directly to the tuning script.
  4. A roadmap for the next actions: The message implicitly defines the next steps: examine the tuning script's input format, determine whether NVFP4 requires special handling, generate SM120-specific configs, restart the server with the new configs, and benchmark the result.

The Thinking Process Visible in Reasoning

The assistant's thinking in message 659 is visible not just in what it says, but in the structure of its actions. The three bash commands are dispatched in parallel, demonstrating a systematic diagnostic approach:

Mistakes and Incorrect Assumptions

The most significant incorrect assumption in this message is the belief that tuning Triton MoE kernel configs will improve performance for the flashinfer_cutlass backend. As we see in subsequent messages (msg 660-662), the server log does not contain the expected "Using default MoE" warning, and the assistant eventually discovers that the flashinfer_cutlass backend uses a completely different kernel path (CUTLASS-based, not Triton-based). The Triton MoE config files at fused_moe_triton/configs/ are irrelevant when the server is using --moe-runner-backend flashinfer_cutlass.

This is a subtle but critical misunderstanding. SGLang has multiple MoE execution backends: a Triton-based one (which reads configs from the fused_moe_triton/configs/ directory) and a flashinfer-based one (which uses CUTLASS kernels from the FlashInfer library). The server was launched with --moe-runner-backend flashinfer_cutlass, meaning it was already using the flashinfer path. The assistant assumed that tuning Triton configs would help, but the actual bottleneck was in the flashinfer CUTLASS MoE autotune system, which had a separate mechanism for enabling SM120 support.

This mistake is understandable. The FINDINGS.md document, which the assistant has been relying on, was written for a Kimi K2 deployment that used the Triton MoE backend. The tuning instructions in that document were specific to the Triton path. The assistant generalized them to the current deployment without fully verifying that the kernel paths matched.

The Broader Significance

Message 659 matters because it represents the moment when the assistant commits to deep kernel optimization. Before this message, the approach was architectural and configurational: switch to LXC, fix CUDA initialization, adjust server parameters, run benchmarks at different concurrency levels. After this message, the approach becomes surgical: modify source code, patch kernel autotune lists, generate hardware-specific configurations, and ultimately fork both SGLang and FlashInfer to add SM120 support.

The message also reveals a fundamental tension in deploying large language models on non-datacenter hardware. The RTX PRO 6000 is a professional GPU with impressive specs (96GB VRAM, 600W TDP), but it uses the SM120 architecture, which is a consumer variant of Blackwell. The entire inference software stack — SGLang, FlashInfer, Triton, TRT-LLM — was primarily developed and tested on datacenter GPUs (H100's SM90, B200's SM100). SM120 support is an afterthought, with missing kernel configs, disabled features, and untested code paths. The assistant's journey in this session is essentially a porting effort: taking software designed for one architecture and making it work on another.

Conclusion

Message 659 is a pivot point disguised as a routine diagnostic. In three bash commands and one sentence of analysis, the assistant synthesizes the model's architectural parameters, assesses the tuning infrastructure, and sets a course for the most impactful optimization available. The assumptions it makes are reasonable given the available information, even though one of them (the relevance of Triton configs to the flashinfer_cutlass backend) will prove incorrect. This message exemplifies the kind of systematic, hypothesis-driven debugging that characterizes effective ML engineering: observe a performance plateau, form a hypothesis about the bottleneck, gather evidence to test the hypothesis, and commit to a course of action. The story of GLM-5 deployment on SM120 Blackwell is, in many ways, the story of this single decision and where it led.