The Turning Point: Diagnosing GPU Underutilization and Initiating MoE Kernel Tuning for GLM-5 on Blackwell
Introduction
In the long arc of deploying the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 658] represents a critical inflection point. It is the moment where the assistant—after rounds of environment setup, driver debugging, crash resolution, and baseline benchmarking—finally confronts the core performance bottleneck head-on. The message is deceptively brief: a single bash command to extract MoE (Mixture-of-Experts) parameters from the model configuration, paired with a file read of a research document. But beneath this surface simplicity lies a sophisticated chain of reasoning about GPU architecture, kernel scheduling, and the fundamental mismatch between a model designed for datacenter hardware and the consumer-grade Blackwell silicon it now runs on.
This article examines message [msg 658] in depth: what prompted it, what decisions it embodies, what knowledge it presupposes, and what it reveals about the assistant's thinking process. It is a message about diagnosis and preparation—the quiet work that precedes any successful optimization.
The message itself reads as follows:
Good observation — low power (~83W idle) and the throughput plateau suggest we're not saturating the GPUs. MoE kernel tuning is the clear next step. Let me check the GLM-5 MoE parameters and get the tuning started.
>
[bash] ssh root@[REDACTED] "python3 -c \" import json with open('/shared/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/config.json') as f: c = json.load(f) for k in ['num_experts', 'num_experts_per_tok', 'moe_intermediate_size', 'hidden_size', 'num_hidden_layers', 'first_k_dense_replace', 'n_routed_experts', 'num_local_experts', 'intermediate_size']: print(f'{k}: {c.get(k)}') \""
>
num_experts: None num_experts_per_tok: 8 moe_intermediate_size: 2048 hidden_size: 6144 num_hidden_layers: 78 first_k_dense_replace: 3 n_routed_experts: 256 num_local_experts: None intermediate_size: 12288
>
[read] /home/theuser/glm-kimi-sm120-rtx6000bw/FINDINGS.md
This short exchange contains a wealth of diagnostic reasoning, architectural knowledge, and strategic decision-making.
The Context That Demanded Action
To understand why this message exists, one must understand the conversation that precedes it. By message [msg 658], the assistant has already accomplished a remarkable amount. It has deployed the GLM-5-NVFP4 model—a massive 256-expert MoE language model with approximately 768GB of weights—across eight RTX PRO 6000 Blackwell GPUs using SGLang. It has resolved NaN crashes during decode by selecting the correct attention backends. It has migrated from a Proxmox VM to an LXC container to bypass virtualization-induced PCIe P2P bottlenecks. It has achieved a working inference server that can serve requests.
But the performance numbers tell a troubling story. At message [msg 655], the assistant benchmarks 256 concurrent requests and gets 879.87 total tokens per second. At message [msg 656], 512 concurrent requests yields only 912.70 tok/s—barely an improvement. The throughput has plateaued, and worse, the GPUs are idling at approximately 83W each, a tiny fraction of their 600W thermal design power (TDP). The user's observation at [msg 657] crystallizes the problem: "Note, still quite low power and lowish pcie bw, probably want to tune kernels."
This is the immediate trigger for message [msg 658]. The assistant is responding to a clear signal: the hardware is dramatically underutilized, and the bottleneck is not network bandwidth or memory capacity but kernel efficiency. The GPUs are not being fed enough work to keep their compute units busy.
The Reasoning: Why MoE Kernel Tuning Specifically?
The assistant's first sentence in the message reveals its reasoning: "Good observation — low power (~83W idle) and the throughput plateau suggest we're not saturating the GPUs. MoE kernel tuning is the clear next step."
This is not an obvious conclusion to someone unfamiliar with modern transformer inference. The assistant could have pursued many other avenues: increasing batch size further, adjusting the number of decode steps, changing the tensor parallelism configuration, or even questioning the model quantization format. Instead, it zeroes in on MoE kernel tuning. Why?
The answer lies in the architecture of GLM-5. The model has 256 routed experts (n_routed_experts: 256), with 8 experts activated per token (num_experts_per_tok: 8). Each expert has an intermediate size of 2048 (moe_intermediate_size: 2048). This means every single token must pass through a gating mechanism that selects 8 out of 256 experts, then compute through those experts' feed-forward networks. The MoE computation dominates the forward pass—it is where the vast majority of FLOPs are spent.
Moreover, the assistant knows from the prior research documented in FINDINGS.md that MoE kernel tuning was the single biggest performance win for a similar model (Kimi K2) on the same hardware. The file read at the end of message [msg 658] pulls up content about "SGLang PR #16975 - SM120 MXFP4 Support," which is described as "MOST PROMISING." This PR specifically addresses SM120 (Blackwell consumer architecture) support for the fused MoE kernels that SGLang uses.
The assistant is thus connecting several dots: low GPU utilization → MoE is the dominant computation → MoE kernels on SM120 are known to be suboptimal → prior work on similar hardware shows large gains from tuning → there is an open PR that specifically addresses this. The decision to pursue MoE kernel tuning is not a guess; it is a reasoned inference from architectural knowledge, prior benchmarks, and community research.
The Decisions Made (and Not Made)
Message [msg 658] is primarily a diagnostic and preparatory message, but it contains several implicit decisions:
Decision 1: Prioritize MoE tuning over other optimizations. The assistant could have continued increasing concurrency, tried TP4+PP2 (tensor parallelism 4 + pipeline parallelism 2), or investigated the PCIe bandwidth more deeply. Instead, it commits to MoE kernel tuning as "the clear next step." This decision is informed by the user's hint and by the assistant's own reading of the research findings.
Decision 2: Extract precise MoE parameters before tuning. The assistant does not assume the model's MoE configuration. It explicitly queries the config.json file to get n_routed_experts, num_experts_per_tok, moe_intermediate_size, hidden_size, and other parameters. This is crucial because the tuning scripts need exact dimensions to generate optimal kernel configurations. A wrong parameter would produce useless tuning results.
Decision 3: Read the existing research rather than start from scratch. The assistant reads FINDINGS.md rather than diving directly into code changes. This reflects a methodical approach: understand what has been tried before, what worked, and what didn't, before committing engineering effort.
Decision 4: Use the correct config file path. Note that the assistant had to find the correct snapshot hash (6944a23f9ffb9668ac970901526c6cc72c34f4e2) because earlier attempts at message [msg 655] failed with a glob expansion issue. The assistant learns from that failure and uses the explicit path this time.
Assumptions Embedded in the Message
Every technical message rests on assumptions, and [msg 658] is no exception:
Assumption 1: The MoE kernel is the primary bottleneck. The assistant assumes that once MoE kernels are properly tuned for SM120, GPU utilization will rise and throughput will increase substantially. This is a reasonable assumption given the model architecture and prior evidence, but it is not proven. The bottleneck could also be in the attention mechanism, the allreduce communication between GPUs, or the CPU-side scheduler.
Assumption 2: The tuning methodology from Kimi K2 transfers to GLM-5. The assistant is about to apply tuning approaches developed for a different model (Kimi K2) to GLM-5. While both are MoE models, their exact expert counts, intermediate sizes, and layer configurations differ. The assistant implicitly assumes that the tuning infrastructure (the tuning_fused_moe_triton.py scripts and config file format) will work for GLM-5's parameters.
Assumption 3: The flashinfer_cutlass backend is the right target. At this point in the conversation, the server is running with --moe-runner-backend flashinfer_cutlass. The assistant assumes that tuning the MoE kernels for this backend will yield improvements, rather than switching to a different backend entirely.
Assumption 4: Low power draw equals underutilization. The assistant interprets ~83W per GPU as evidence that the GPUs are not being fully utilized. While this is generally correct, it is worth noting that Blackwell GPUs can show low power during memory-bound workloads even when the compute units are busy. The assistant's inference is sound but not absolute.
Input Knowledge Required
To fully understand message [msg 658], a reader needs knowledge spanning several domains:
Model Architecture Knowledge: Understanding what MoE (Mixture-of-Experts) means, why n_routed_experts: 256 with num_experts_per_tok: 8 matters, and how the intermediate size affects computation. The reader must know that MoE layers replace the standard feed-forward network with multiple "expert" sub-networks, and that a gating mechanism selects a subset of experts per token.
GPU Architecture Knowledge: Understanding SM120 (Blackwell) and why it differs from SM90 (Hopper) or SM100 (datacenter Blackwell). The RTX PRO 6000 uses a consumer Blackwell die (SM120) which has different shared memory sizes, different tensor core capabilities, and different optimal kernel configurations compared to the datacenter B200 (SM100).
SGLang Runtime Knowledge: Understanding that SGLang uses pre-tuned kernel configuration files stored in python/sglang/srt/layers/moe/fused_moe_triton/configs/, and that the server selects a config file based on expert count (E), intermediate size (N), and device name. Without this knowledge, the assistant's file reads and parameter extraction would seem disconnected.
Quantization Knowledge: Understanding NVFP4 (NVIDIA FP4 quantization) and why it requires specialized kernel support. The model weights are stored in 4-bit floating point format, which means the MoE kernels must handle non-standard data types.
Prior Session History: The reader must know about the Kimi K2 tuning work documented in FINDINGS.md, the PR #16975 for SM120 support, and the fact that the assistant has already tried various attention backends and server configurations.
Output Knowledge Created
Message [msg 658] produces several concrete outputs:
1. Confirmed MoE Parameters: The assistant now knows definitively that GLM-5 has E=256 experts, N=2048 intermediate size, top_k=8, hidden_size=6144, and 78 layers with the first 3 being dense (not MoE). These parameters are the input to any tuning script.
2. Research Context Refreshed: The assistant has re-read the relevant portion of FINDINGS.md, reminding itself of the SM120 MXFP4 support PR and the tuning methodology used for Kimi K2. This knowledge will inform the next steps.
3. Diagnostic Signal Validated: The assistant has confirmed that the low power observation is real and that the throughput plateau at ~900 tok/s is not a measurement artifact. This validates the decision to pursue kernel tuning.
4. Path Forward Established: The message sets up the next phase of work. After extracting parameters and reviewing prior research, the assistant is now positioned to either run the tuning scripts or apply pre-tuned configurations.
The Thinking Process Revealed
The assistant's reasoning in message [msg 658] is visible through several signals:
The opening acknowledgment ("Good observation") shows the assistant incorporating user feedback into its model of the problem. It is not defensive or dismissive; it treats the user's observation as valuable diagnostic input.
The phrase "suggest we're not saturating the GPUs" reveals a mental model where GPU utilization is the key metric to optimize. The assistant thinks in terms of saturation—are the compute units fully occupied?—rather than raw throughput numbers.
The phrase "MoE kernel tuning is the clear next step" shows confidence in the diagnosis. The assistant does not say "let's try MoE tuning" or "MoE tuning might help." It says "the clear next step," indicating a high degree of certainty about the causal chain: low utilization → MoE bottleneck → tuning fixes it.
The parallel execution of bash and read shows the assistant working efficiently. It extracts model parameters while simultaneously reading research notes. This is not a linear thinker; it is a systems thinker who understands that information gathering can be parallelized.
The careful parameter extraction (requesting specific keys from config.json) shows attention to detail. The assistant does not just read the whole config file; it extracts precisely the MoE-relevant fields. It knows exactly what it needs.
The file read targeting FINDINGS.md shows that the assistant values prior work. It does not assume it remembers everything from earlier in the conversation; it re-reads the source material to ensure accuracy.
Mistakes and Incorrect Assumptions
While message [msg 658] is technically sound, it contains one notable limitation: the assistant assumes that MoE kernel tuning alone will resolve the underutilization. In reality, the problem is multi-faceted. As later messages in the conversation reveal, the flashinfer allreduce fusion is unsupported on SM120, which means inter-GPU communication during the allreduce step is inefficient. The MoE kernel tuning improves throughput dramatically (from ~880 to ~3,740 tok/s in the next chunk), but GPU power draw remains around 250W out of 600W TDP. The assistant's assumption that MoE tuning is sufficient turns out to be incomplete—it is necessary but not sufficient.
Additionally, the assistant does not yet question whether the flashinfer_cutlass backend is the right choice for SM120. It proceeds with tuning under the assumption that this backend is optimal, when in fact the flashinfer_trtllm backend might offer different trade-offs. This assumption is corrected later in the conversation when the assistant experiments with alternative backends.
Conclusion
Message [msg 658] is a masterclass in diagnostic reasoning. It takes a vague symptom—low GPU power draw—and traces it through architectural knowledge to a specific root cause: suboptimal MoE kernels for the SM120 architecture. It extracts precise parameters, consults prior research, and establishes a clear path forward. The message is brief, but it represents the convergence of many threads: hardware knowledge, model architecture understanding, runtime system familiarity, and a methodical approach to performance optimization.
In the broader narrative of deploying GLM-5 on Blackwell, this is the moment where the assistant stops fighting fires (crashes, driver issues, virtualization problems) and starts optimizing for performance. It is the pivot from "making it work" to "making it fast." The subsequent chunk shows that this pivot was successful—throughput jumps to nearly 4,000 tok/s—but also that the optimization journey is never truly complete. The assistant's willingness to diagnose, question assumptions, and iterate is what makes this session a compelling case study in real-world ML infrastructure engineering.