The MoE Kernel Tuning Hypothesis: A Pivot in Debugging GLM-5-NVFP4 Throughput

In the middle of a complex debugging session spanning multiple days, a single message from the assistant at index 273 marks a critical turning point. The conversation had been wrestling with deploying the massive GLM-5-NVFP4 model — a 744-billion-parameter Mixture-of-Experts (MoE) model — across eight NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. After resolving a crippling NaN crash during decode by selecting the correct attention backends, the team had turned to performance tuning. But throughput remained stubbornly low: roughly 10–11 tokens per second for single requests, scaling to around 200–250 tokens per second with high concurrency. The GPUs were running at 100% utilization yet drawing only 55% of their 600W power budget, suggesting the workload was compute-bound but failing to saturate the tensor cores.

Then the user dropped a cryptic hint: "./ did some weird tuning configs, use an expert to figure out what that was exactly @README.md" ([msg 270]). This referred to a local research repository containing artifacts from a prior deployment of Kimi K2 models on the same hardware. The assistant dispatched a subagent to investigate ([msg 272]), and the task result returned a detailed analysis of MoE kernel tuning configurations. Message 273 is the assistant's response to that task result — and it represents a fundamental reframing of the performance problem.

The Reasoning: Connecting Prior Art to a New Hypothesis

The message opens with a moment of synthesis: "Very useful. The prior Kimi K2 work used tuning_fused_moe_triton.py to generate hardware-specific MoE kernel configs for the RTX PRO 6000." This is the key insight. The assistant recognizes that the research repository contains not just notes but actual tuning infrastructure — a script that benchmarks different block sizes and kernel configurations to find the optimal settings for a specific GPU and model combination.

But then comes the critical observation: "The existing configs are for GLM-4.7-FP8 (E=161, N=192/384) — completely different dimensions from GLM-5 (E=256, N=256 at TP8)." The variables here are crucial: E is the number of experts in the MoE layer, and N is the intermediate size per expert (after tensor parallelism sharding). GLM-4.7-FP8 has 161 experts with N=192 or N=384 depending on TP configuration, while GLM-5-NVFP4 has 256 experts with N=256 at TP8. These are fundamentally different matrix dimensions, which means the optimal Triton kernel block sizes for one model would be entirely inappropriate for the other.

The assistant then articulates the core hypothesis: "Without a tuned config, sglang falls back to generic defaults which are likely terrible for this GPU + model combination. That would explain low GPU efficiency — the MoE Triton kernels are using suboptimal block sizes." This is a powerful diagnostic leap. Earlier in the conversation ([msg 269]), the assistant had concluded the bottleneck was "kernel launch overhead and small matmul efficiency" — but now there's a specific, testable mechanism for why the kernels are inefficient. The generic defaults in sglang's fused MoE Triton kernels are not tuned for the Blackwell SM120 architecture combined with GLM-5's specific expert dimensions.

The Decision: Investigate Before Acting

The assistant does not immediately jump to running the tuning script. Instead, it takes a measured approach: "Let me check if sglang is complaining about a missing config, and then run the tuning." This is a two-step plan. First, verify that sglang is indeed falling back to defaults (by looking for warning messages in the server log). Second, run the tuning script to generate proper configurations.

The bash command searches the sglang server log for patterns like "moe.config", "fused_moe", "kernel config", "tuning", and "not found.config". The output shows only a warning about Transformers version compatibility and the server arguments — no explicit complaint about a missing MoE config. This is informative in itself: sglang silently uses defaults without warning when no tuned config is found. The absence of an error message doesn't mean the config is optimal; it means the fallback path is silent.

Input Knowledge Required

To fully appreciate this message, one must understand several layers of context:

MoE Architecture: Mixture-of-Experts models route each token through a subset of "expert" feedforward networks. The router selects the top-k experts, and the token is processed by those experts' weight matrices. The dimensions E (number of experts) and N (intermediate hidden size) determine the shape of these matrix multiplications.

Triton Kernels: OpenAI's Triton language is used to write high-performance GPU kernels. For MoE models, the fused_moe_triton kernel fuses the expert computation into a single kernel launch, avoiding overhead from launching many small kernels. However, Triton kernels require tuning parameters like block sizes (e.g., BLOCK_SIZE_M, BLOCK_SIZE_N, BLOCK_SIZE_K) that depend on the specific matrix dimensions and GPU architecture.

sglang's MoE Runner: sglang supports multiple backends for executing MoE layers, including triton, flashinfer_cutlass, and flashinfer_trtllm. The Triton backend uses tuning_fused_moe_triton.py to benchmark and select optimal block sizes, storing results in JSON config files.

The Prior Work: The research repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ contains artifacts from deploying Kimi K2.5 and K2-Thinking models on the same 8x RTX PRO 6000 Blackwell hardware. This includes MoE tuning configs generated by the tuning script, which were essential for getting reasonable performance on those models.

GPU Utilization Context: Earlier in the session ([msg 267]), the assistant had discovered all 8 GPUs were at 100% utilization but only drawing ~328W out of 600W TDP — just 55% of the power budget. This indicated the GPUs were compute-bound but not fully utilizing their tensor cores, consistent with inefficient small-matrix operations.

Output Knowledge Created

This message produces several important outputs:

  1. A new diagnostic hypothesis: The low GPU efficiency may be caused by suboptimal MoE Triton kernel configurations, not just inherent limitations of small-batch decode.
  2. A concrete action plan: Check the server log for config warnings, then run the tuning script to generate GLM-5-specific configurations.
  3. A connection between prior and current work: The Kimi K2 research artifacts contain reusable infrastructure (the tuning script) that can be directly applied to GLM-5.
  4. A reframing of the bottleneck analysis: What appeared to be an unavoidable compute-bound limitation might be a solvable configuration problem.

Assumptions and Potential Pitfalls

The assistant makes several assumptions worth examining:

That sglang is using the Triton MoE runner: The server was launched with --moe-runner-backend flashinfer_cutlass (visible in the next message, [msg 274]), meaning it's using the flashinfer+cutlass backend, not the Triton fused_moe backend that the tuning configs target. The assistant hasn't yet verified which backend is active. This is a significant assumption — if the server isn't using Triton kernels at all, the tuning configs are irrelevant.

That generic defaults are "likely terrible": This is a reasonable assumption given the GPU utilization data, but it's not yet proven. The tuning script would need to be run and compared to establish causation.

That the tuning script will work for GLM-5: The script was designed for GLM-4.7-FP8's dimensions. It may need modification to handle E=256 and N=256, or it may be generic enough to work with any dimensions.

That the tuning configs from prior work are the primary lever: There could be other factors limiting throughput — the virtualization overhead from Proxmox, PCIe topology limitations, or the NSA attention backend overhead — that the MoE kernel tuning won't address.

The Thinking Process Visible in Reasoning

The message reveals a structured analytical mind at work. The assistant doesn't just accept the task result at face value; it actively synthesizes the information with prior knowledge. The phrase "That would explain low GPU efficiency" shows the assistant connecting the new information (existence of tuning configs for different model dimensions) with the observed symptoms (100% GPU utilization but low power draw and throughput).

The decision to check the server log before running the tuning script demonstrates disciplined debugging: verify the hypothesis is relevant before investing time in the solution. The specific grep patterns chosen — "moe.config", "fused_moe", "kernel config", "tuning", "not found.config" — show an understanding of what sglang's logging infrastructure would emit in different scenarios.

The message also shows the assistant working within the constraints of the remote environment. It uses ssh to access the server machine, greps through log files, and plans to run the tuning script remotely. This is a session where the assistant has no direct filesystem access to the target machine — everything goes through SSH, adding latency and complexity to every operation.

Broader Significance

This message represents a classic moment in systems debugging: the shift from "we've hit a fundamental limit" to "we might have a configuration problem." The assistant had previously concluded that the throughput was limited by the physics of small-batch decode on large MoE models — a conclusion that, if accepted, would close the door on further optimization. The user's hint about the tuning configs reopened that door, and the assistant's analysis in message 273 turned a vague clue into a specific, actionable hypothesis.

The message also illustrates the value of maintaining research artifacts from prior work. The Kimi K2 repository, built during a previous deployment on the same hardware, contained exactly the infrastructure needed to debug the current problem. Without it, the assistant might have continued down the path of accepting the compute-bound diagnosis as final.

In the messages that follow ([msg 274] onward), the assistant discovers that sglang is actually using the flashinfer_cutlass backend, not the Triton backend, which means the tuning configs from prior work don't directly apply. But the investigation doesn't stop there — the assistant pivots to examining what the flashinfer backend is doing and whether it has its own tuning mechanisms. The core insight from message 273 — that MoE kernel configuration matters and the current setup may be suboptimal — continues to drive the investigation forward, even as the specific mechanism changes.