The Comparative Epiphany: Mining Prior Success to Unlock GLM-5 Inference Performance

In the middle of a high-stakes optimization session for deploying the GLM-5-NVFP4 model on 8x NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant has a moment of recognition. Message [msg 669] captures a critical turning point: the assistant spots a "key difference" between the current deployment configuration and a prior successful run of the Kimi K2-Thinking model on identical hardware. This seemingly small observation—that the prior run used moe_runner_backend='auto', fp4_gemm_runner_backend='auto', attention_backend='triton', and disable_cuda_graph=True—represents a fundamental shift in debugging strategy. Instead of continuing to tweak parameters in isolation, the assistant pivots to comparative analysis, using a known-working configuration as a Rosetta Stone to decode why the current setup is underperforming.

Context: The Performance Plateau

To understand the significance of this message, one must appreciate the frustration that precedes it. The assistant has been engaged in an extended optimization campaign spanning multiple segments of the conversation. The GLM-5-NVFP4 model—a massive Mixture-of-Experts (MoE) architecture with 256 experts, 78 layers, and a hidden size of 6144—has been deployed on eight RTX PRO 6000 Blackwell GPUs (SM120 architecture) using tensor parallelism of 8. Despite successful deployment and the resolution of earlier NaN crashes during decode, throughput has plateaued at approximately 880–912 total tokens per second. GPU power draw hovers around 83W per GPU, a fraction of the 600W thermal design power (TDP). The user's observation in [msg 657]—"still quite low power and lowish pcie bw, probably want to tune kernels"—crystallizes the problem: the hardware is dramatically underutilized.

The assistant's initial response in [msg 658] is to investigate MoE kernel tuning, checking model configuration parameters (E=256 experts, N=2048 intermediate size, top_k=8) and exploring the sglang codebase for tuning infrastructure. But by [msg 664], a new approach emerges: the user directs the assistant to examine a local repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ containing research artifacts from a prior deployment of Kimi K2 and K2-Thinking models on the exact same GPU hardware. This repository becomes a goldmine of comparative data.

The Epiphany: What "Auto" Actually Means

Message [msg 669] opens with a moment of synthesis: "Now I see the key difference." The assistant has been reading the K2-Thinking final benchmark log and has identified four configuration parameters that distinguish the successful prior run from the current GLM-5 deployment:

  1. moe_runner_backend='auto' — Instead of explicitly selecting flashinfer_cutlass (which the current GLM-5 server was using), the K2-Thinking run allowed sglang to automatically resolve the best MoE runner backend for the hardware and quantization format.
  2. fp4_gemm_runner_backend='auto' — Similarly, the FP4 GEMM (General Matrix Multiply) runner backend was left to automatic resolution, rather than being forced to a specific implementation.
  3. attention_backend='triton' — The attention backend was explicitly set to Triton, which the log shows was the default anyway ("Attention backend not specified. Use triton backend by default").
  4. disable_cuda_graph=True — CUDA graph capture was disabled, which is significant because CUDA graphs can introduce overhead for dynamic MoE routing patterns. The critical insight here is that "auto" is not a cop-out—it is an informed delegation to sglang's backend resolution logic, which considers hardware architecture (SM120), quantization format (modelopt_fp4/NVFP4), and available kernel implementations to select the optimal path. By forcing flashinfer_cutlass explicitly, the current GLM-5 deployment may have been overriding a more intelligent selection.

The Investigation Method: Forensic Log Mining

The assistant's approach in this message is a textbook example of forensic performance analysis. Rather than restarting the server with new parameters and re-benchmarking (a time-consuming process), the assistant mines the existing K2-Thinking log for clues. The grep command issued is meticulously crafted:

grep -n 'autotun\|AutoTun\|tactic\|flashinfer.*cutlass\|moe_runner\|fp4_gemm\|FP4\|backend\|warmup.*start\|warmup.*end\|warmup.*autotune\|Benchmark\|Total token\|Output token\|Peak output' /home/theuser/glm-kimi-sm120-rtx6000bw/logs/sglang-k2-thinking-final-bench.log | head -60

This pattern targets multiple categories of information simultaneously:

Assumptions and Their Implications

The assistant operates under several assumptions in this message, some explicit and some implicit:

Assumption 1: The K2-Thinking configuration is transferable to GLM-5. This is the central premise of the comparative approach. The assistant assumes that because both models use NVFP4 quantization and run on the same SM120 hardware with the same tensor parallelism (TP8), the backend resolution that worked for K2-Thinking will also work for GLM-5. This is a reasonable assumption but not guaranteed—GLM-5 has different MoE parameters (E=256, N=2048 vs K2-Thinking's E=161, N=192 based on the moe-configs directory), which could affect backend suitability.

Assumption 2: "Auto" resolved to something specific and discoverable. The assistant expects that the log will contain evidence of what moe_runner_backend='auto' and fp4_gemm_runner_backend='auto' actually resolved to. However, the grep output shown doesn't yet reveal this information—the log lines are truncated. The assistant may need to search more specifically for resolution messages.

Assumption 3: The prior run was successful and well-configured. The assistant treats the K2-Thinking deployment as a reference implementation. This is validated by the existence of benchmark reports (report-kimi-k2-sm120.html, report-kimi-k2-sm120.pdf) and the README's claim of "Successfully deployed large MoE models." However, the assistant hasn't yet confirmed the throughput numbers from that run—the grep targeted benchmark result lines but the output shown doesn't include them.

Assumption 4: Backend selection is the primary bottleneck. The assistant implicitly assumes that the MoE runner backend and FP4 GEMM backend choices are the dominant factors limiting throughput, rather than, say, PCIe bandwidth, memory bandwidth, or the attention kernel. This assumption is informed by the user's hint about kernel tuning and the low GPU power draw, which suggests the compute units are not being fully utilized.

Knowledge Required and Created

Input knowledge required to understand this message includes: familiarity with sglang's server architecture and backend resolution system; understanding of MoE inference and the role of runner backends; awareness of the NVFP4 quantization format and its implications for GEMM operations; knowledge of CUDA graphs and their impact on dynamic execution; and context about the prior Kimi K2 research artifacts stored locally.

Output knowledge created by this message is primarily diagnostic: the assistant has identified a concrete configuration discrepancy between the current and prior deployments. This creates a testable hypothesis: "If we restart the GLM-5 server with moe_runner_backend='auto', fp4_gemm_runner_backend='auto', and disable_cuda_graph=True, we may see improved throughput and GPU utilization." The message also surfaces the specific log file and grep patterns needed to extract backend resolution decisions, establishing a methodology for further investigation.

The Thinking Process: A Window into Debugging Strategy

The reasoning visible in this message reveals a sophisticated debugging approach. The assistant has moved through several phases:

  1. Direct optimization (messages [msg 655]–[msg 662]): Attempting to tune MoE kernels directly by examining config files, running benchmarks, and checking server logs.
  2. Architecture investigation ([msg 662]): Using a subagent task to trace the MoE kernel code path for the flashinfer_cutlass backend with NVFP4 quantization.
  3. Comparative analysis ([msg 664]–[msg 669]): Pivoting to examine prior work on identical hardware when direct optimization proves insufficient. The "key difference" realization in [msg 669] is the payoff of this comparative approach. The assistant doesn't just note the difference—it immediately formulates the next investigative step: "Let me check what 'auto' resolved to for FP4 MoE on that run and see the benchmark numbers." This demonstrates a disciplined debugging mindset: having found a configuration difference, the assistant seeks to understand the mechanism by which that difference produced better results, not just blindly copy the parameters. The truncated grep output is itself revealing. The assistant used head -60, which limits the results to the first 60 matching lines. This suggests the assistant expected the log to be large and wanted to focus on the earliest occurrences—likely the server startup phase where backend resolution happens. The output shows server args (line 6) and the attention backend default message (line 3), but the MoE-specific resolution messages may appear later in the log or may not match the grep pattern.

Broader Significance

This message represents a methodological turning point in the optimization session. Prior to [msg 669], the assistant was operating in a relatively isolated manner, treating the GLM-5 deployment as a novel problem to be solved from first principles. The discovery of the local research repository and the comparative analysis it enables transforms the problem from "how do we optimize GLM-5 on SM120?" to "what did we learn from optimizing K2-Thinking on SM120, and how does it apply here?"

This is a common pattern in complex engineering work: the most valuable insights often come not from exhaustive parameter sweeps, but from recognizing patterns across related efforts. The assistant's willingness to pause the direct optimization and dive into prior artifacts demonstrates a meta-cognitive awareness that the current approach was hitting diminishing returns.

The message also highlights the importance of maintaining detailed logs and research artifacts. The existence of the glm-kimi-sm120-rtx6000bw repository with its structured directories (configs/, logs/, moe-configs/, patches/, source/) and comprehensive README enables exactly this kind of cross-project learning. Without it, the assistant would be forced to rediscover optimal configurations through trial and error.

What Comes Next

Message [msg 669] sets the stage for a critical experiment. The assistant has identified the configuration parameters to investigate and has begun extracting data from the reference log. The next logical steps would be: (1) determine what "auto" resolved to for each backend in the K2-Thinking run, (2) extract the benchmark throughput numbers from that run to quantify the performance gap, (3) restart the GLM-5 server with matching parameters, and (4) re-benchmark to measure the improvement.

The message also implicitly raises a deeper question: if the backends were the bottleneck, why did the explicit flashinfer_cutlass selection underperform the auto-resolved choice? Possible explanations include: the auto-resolution selected a different CUTLASS kernel variant better suited to SM120's smaller shared memory; the auto-resolution enabled additional fusion optimizations; or the explicit selection inadvertently disabled fallback paths that auto-resolution would use. Answering this question would require tracing through sglang's backend resolution logic—a task the assistant's earlier subagent investigation ([msg 662]) had already begun.

In the broader narrative of the session, this message is the hinge point between frustration and breakthrough. The throughput numbers that follow in subsequent messages—jumping from ~880 tok/s to ~3,740 tok/s—will validate the comparative approach and demonstrate that sometimes the fastest path to optimization is learning from those who have already walked it.