The Critical Observation That Changed the Trajectory of GLM-5 Inference Tuning

"Note, still quite low power and lowish pcie bw, probably want to tune kernels"

In the middle of an intensive session to deploy the massive GLM-5-NVFP4 mixture-of-experts model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single user message at index 657 cut through the noise of benchmark numbers and redirected the entire optimization effort. This brief, almost casual observation—just fourteen words—carried more diagnostic weight than the preceding pages of throughput figures and server configurations. It was the moment when the conversation shifted from "how many tokens can we push through" to "why aren't the GPUs actually working?"

The Context: A Plateau Hidden Behind Numbers

To understand why this message was so pivotal, we need to examine what had just happened. The assistant, following a well-structured plan laid out in [msg 652], had been systematically testing higher concurrency levels to find the throughput ceiling of the GLM-5 deployment. The logic was straightforward: increase the number of concurrent requests, push more tokens through the pipeline, and find where the system saturates.

The benchmarks told a story that looked like progress. At 256 concurrent requests ([msg 655]), the system achieved 879.87 total tokens per second. At 512 concurrent requests ([msg 656]), it reached 912.70 total tokens per second. The assistant's own commentary reflected cautious optimism: "getting closer to 1k!" But these numbers masked a deeper problem. The throughput had barely budged despite doubling the request load—a classic sign of hitting a bottleneck that isn't about request scheduling.

Meanwhile, the GPU monitoring data from [msg 653] painted a damning picture: every GPU was drawing approximately 85 watts of power, with 0% utilization reported. These are RTX PRO 6000 Blackwell cards with a 600-watt thermal design power (TDP) each. During active inference serving, they were consuming barely more than idle power. The GPUs were essentially asleep.

The User's Insight: Reading Between the Lines of Benchmark Data

The user's message in [msg 657] synthesized two critical observations from the available data:

"still quite low power" — This acknowledged that despite the server running and serving requests, the GPUs were not being exercised. The 85-watt power draw was a telltale sign that the compute cores were largely idle. In any GPU-accelerated workload, power draw is a proxy for actual computation happening. When eight GPUs collectively drawing less than 700 watts are supposed to be running a 256-expert MoE model with tensor parallelism across all of them, something fundamental is wrong. The model's MoE layers—which should be the most compute-intensive part of inference—were apparently not engaging the hardware effectively.

"lowish pcie bw" — This observation pointed to the interconnect between GPUs. In a tensor-parallel deployment, the GPUs must constantly exchange intermediate results. Low PCIe bandwidth utilization suggests that either the communication pattern is inefficient, or the compute phases are so short that the GPUs spend most of their time waiting on synchronization rather than doing useful work. Both point to the same root cause: the GPU kernels being used were not optimized for this architecture.

The user's conclusion—"probably want to tune kernels"—was not a guess. It was a diagnosis. The symptoms (low power, low bandwidth utilization, throughput plateauing despite increasing concurrency) all pointed to inefficient kernel execution rather than insufficient request pressure.

The Reasoning Process: What Made This Diagnosis Possible

The user's ability to make this diagnosis depended on several pieces of knowledge and reasoning:

1. Understanding GPU power as a performance metric: Power draw during inference is not just an operational concern—it is a direct signal of compute utilization. A GPU drawing 85W while serving requests is not doing meaningful computation. The user implicitly understood that the gap between 85W and 600W TDP represented untapped performance potential.
2. Recognizing the architecture mismatch: The GLM-5 model uses a massive MoE architecture with 256 experts, 8 experts active per token, and an intermediate size of 2048 per expert (as confirmed in [msg 658]). The RTX PRO 6000 is a Blackwell SM120 architecture, which differs significantly from the datacenter Blackwell SM100 that most inference frameworks target. The user understood that kernel tuning is architecture-specific and that off-the-shelf kernels might not be optimized for SM120.
3. Interpreting throughput saturation: The move from 256 to 512 concurrency yielded only a ~3.7% throughput improvement (879 to 912 tok/s). This saturation pattern—where doubling request load produces negligible gains—is characteristic of a compute-bound bottleneck, not a request-scheduling bottleneck. The user correctly identified that more concurrency would not solve the problem.
4. Knowledge of the MoE kernel landscape: The user's suggestion to "tune kernels" specifically referenced the mixture-of-experts kernels, which are the most performance-critical component of GLM-5 inference. MoE models require specialized kernels for expert routing, gating, and the sparse matrix operations that select and combine expert outputs. These kernels are often hand-tuned for specific GPU architectures, and SM120 support was known to be immature.

Assumptions Embedded in the Message

The user's message carried several implicit assumptions that shaped how the assistant would respond:

• That kernel tuning was feasible: The user assumed that the existing MoE kernels could be tuned or replaced for SM120 performance. This was not guaranteed—some kernel optimizations are baked into compiled CUDA code that may not support SM120 at all. The assistant's subsequent exploration would reveal that FlashInfer's allreduce fusion, for example, was explicitly disabled on SM120 because the underlying TRT-LLM communication kernels only supported SM90 and SM100 architectures.
• That the bottleneck was in the compute kernels, not the communication layer: The user attributed the low PCIe bandwidth to kernel inefficiency rather than to fundamental hardware topology limitations. This assumption would later be challenged when the assistant discovered that PCIe P2P bottlenecks in the Proxmox VM environment were a separate, hardware-level constraint.
• That the assistant had the capability to modify and recompile kernels: The suggestion implicitly assumed that the assistant could patch source code, rebuild FlashInfer or SGLang components, and test alternative kernel implementations. This was a reasonable assumption given the assistant's demonstrated ability to modify source code throughout the session.

What Knowledge Was Required to Understand This Message

For a reader to fully grasp the significance of this message, several pieces of background knowledge are necessary:

• GPU power draw as a diagnostic: Understanding that modern GPUs have sophisticated power management and that power draw correlates strongly with compute utilization. An idle GPU typically draws 30-80W; a fully utilized GPU under load draws near its TDP (600W for these cards).
• Tensor parallelism and MoE inference: Knowledge that GLM-5 uses a mixture-of-experts architecture where each token activates only a subset of experts, requiring specialized routing and sparse computation. Tensor parallelism across 8 GPUs means each GPU holds a shard of each expert and must communicate partial results.
• SM120 vs SM100 architecture differences: Understanding that NVIDIA's Blackwell architecture has multiple variants (SM100 for datacenter, SM120 for consumer/workstation) with different characteristics like shared memory size, register file configuration, and supported instruction sets. Kernel code compiled for SM100 may not run optimally—or at all—on SM120.
• The role of kernel autotuning: Awareness that frameworks like FlashInfer and SGLang use autotuning to select the best kernel implementation for a given GPU architecture. If SM120 is not in the autotune database, the framework may fall back to generic kernels that perform poorly.

The Impact: How This Message Redirected the Session

The assistant's response in [msg 658] immediately acknowledged the correctness of the observation: "Good observation — low power (~83W idle) and the throughput plateau suggest we're not saturating the GPUs. MoE kernel tuning is the clear next step." This marked a sharp pivot from the previous trajectory of testing higher concurrency levels to a focused investigation of kernel performance.

The subsequent work in the chunk ([msg 650] through [msg 754]) would validate the user's diagnosis dramatically. By enabling FlashInfer CUTLASS MoE autotune for SM120—patching model_runner.py to include flashinfer_cutlass in the autotune list—and raising --max-running-requests from 64 to 1024, the assistant would eventually push throughput from ~880 tok/s to ~3,740 tok/s at 1024 concurrency, with peak output reaching nearly 4,000 tok/s. That is a 4.25x improvement, directly attributable to the kernel tuning that the user identified as necessary.

However, the user's message also contained a subtle limitation. The assumption that kernel tuning alone would solve the low-power problem was only partially correct. Even after the dramatic throughput improvements, GPU power draw remained around 250W out of 600W TDP—better than 85W, but still far from full utilization. The assistant would later discover that FlashInfer's allreduce fusion was disabled on SM120, and attempts to patch it resulted in catastrophic performance degradation (dropping to 236 tok/s and 125W power). This revealed that some bottlenecks were architectural rather than merely configurational.

Conclusion: The Power of a Single Diagnostic Observation

The user message at [msg 657] exemplifies the most valuable kind of intervention in a complex technical debugging session: a concise, data-driven observation that reframes the problem. Rather than prescribing a specific solution, the user provided a diagnosis ("low power and lowish pcie bw") and a direction ("probably want to tune kernels"), trusting the assistant to execute the technical work.

This message worked because it connected observable symptoms (power draw, bandwidth utilization) to their root cause (kernel inefficiency) through a chain of reasoning that required deep understanding of GPU architecture, inference serving dynamics, and the specific characteristics of MoE models. It saved the session from continuing down the unproductive path of chasing higher concurrency numbers and redirected effort toward the actual bottleneck.

The result—a 4.25x throughput improvement—stands as a testament to the power of asking the right question at the right moment. In the world of high-performance ML inference, where a single configuration choice can mean the difference between 800 tok/s and 3,700 tok/s on the same hardware, the ability to read the diagnostic signals correctly is often the difference between success and stagnation.