The Turning Point: How a Single Task Call Uncovered the GPU Utilization Mystery in GLM-5-NVFP4 Inference

The Message

In the middle of a high-stakes debugging session for GLM-5-NVFP4 inference on eight RTX PRO 6000 Blackwell GPUs, the assistant sent this message:

Good, server is still running. Let me now investigate the actual compute path. The key question is: what operations are happening per forward pass and how much of the GPU compute (FP4 tensor cores) are we actually using? [task] {"description":"Analyze GLM-5 forward pass compute","prompt":"I need to understand what's limiting GPU utilization for GLM-5-NVFP4 inference on 8x RTX PRO 6000 (SM120, 600W TDP, 96GB HBM each). During high-concurrency benchmarks, GPUs only reach ~250W/600W (42% TDP) despite showing 100% utilization...."}

On its surface, this appears to be a simple delegation — spawn a subagent to analyze the forward pass. But in the arc of the conversation, this message represents a critical inflection point. It is the moment the assistant stopped throwing configuration changes at the wall and instead asked the fundamental question: what is the GPU actually computing, and why isn't it working harder?

The Crisis That Led Here

To understand why this message was written, we must trace the conversation that preceded it. The session had been a long and arduous journey of deploying GLM-5-NVFP4 — a massive Mixture-of-Experts model quantized to 4-bit floating point (NVFP4) — on a novel hardware target: eight RTX PRO 6000 Blackwell GPUs connected via PCIe in a Proxmox virtualized environment.

The path had been anything but smooth. Earlier segments saw the team wrestling with NaN crashes during decode (resolved by selecting the correct attention backends), battling virtualization-induced PCIe latency, attempting to enable P2P DMA between GPUs (ultimately blocked by hardware topology), and finally escaping to an LXC container for bare-metal-like performance. By segment 6, the assistant had achieved a respectable ~3,740 tok/s total throughput at 1024 concurrency — a significant improvement from the initial ~880 tok/s.

But something was wrong. The GPUs were drawing only ~250W out of a 600W TDP. The user had flagged this repeatedly (messages 723, 725), noting that the RTX PRO 6000 has subtle hardware differences from datacenter Blackwell (B200) — smaller shared memory, different kernel requirements — and suggesting that attention might be a culprit. The user even provided a research document (sm120-attention-fix-research.md) outlining the shared memory constraints of SM120 consumer Blackwell versus SM100 datacenter Blackwell.

The assistant initially responded by running more benchmarks and checking power readings (messages 724, 727-728), confirming the 600W TDP and the ~42% utilization. But these were surface-level observations. The assistant needed to go deeper.

Why This Message Was Written: The Reasoning

The assistant's reasoning in message 729 reflects a crucial shift in strategy. The opening line — "Good, server is still running" — acknowledges that the system is stable enough to investigate. The server isn't crashing anymore; the NaN issues are resolved. Now the question is performance.

The key insight is in the second sentence: "Let me now investigate the actual compute path." This signals a move from black-box benchmarking to white-box analysis. Instead of tweaking server parameters and re-running benchmarks, the assistant decides to trace the actual operations happening inside the GPU during a forward pass.

The question "what operations are happening per forward pass and how much of the GPU compute (FP4 tensor cores) are we actually using?" is the right question to ask. The assistant recognizes that GPU utilization reported as "100%" by tools like nvidia-smi can be misleading — the GPU can be 100% busy doing nothing useful if it's stalled on memory transfers or synchronization. The real metric is whether the FP4 tensor cores — the specialized hardware for 4-bit matrix multiplication — are being fed with work.

This reasoning led the assistant to spawn a subagent task. The task tool in the opencode framework creates a subagent that runs its own multi-round conversation to completion, then returns its findings. This is a deliberate architectural choice: rather than trying to manually trace through the codebase (which spans sglang, flashinfer, vLLM, and custom kernels), the assistant delegates the deep code analysis to a focused subagent that can systematically trace the forward pass.

How the Decision Was Made

The decision to use a subagent task rather than direct investigation was shaped by several factors visible in the conversation history:

Complexity of the codebase: The GLM-5 inference path involves multiple layers of abstraction — the sglang server, the model runner, the MoE dispatcher, attention backends (flashinfer, triton, trtllm), the allreduce communicator, and the underlying CUDA kernels. Tracing this manually would require reading dozens of files.
Previous successful use of tasks: Earlier in the conversation, the assistant had used similar task subagents for analysis (visible in the context). This pattern had proven effective for deep dives.
The need for systematic tracing: The assistant needed to trace the forward pass layer by layer, identifying data types, kernel launches, and communication patterns. This is exactly the kind of methodical investigation that a subagent with a clear prompt can perform well.
Parallelism: The task call in this message runs alongside the assistant's own reasoning. While the subagent executes, the assistant can continue other work (in subsequent rounds), though the parent session blocks until the task returns. The prompt to the subagent was comprehensive: it asked about data types in each stage, the MoE compute path, attention compute path, allreduce patterns, and specific hypotheses about why utilization was low. This structured prompt reflects the assistant's understanding of what questions needed answering.

Assumptions Made

Several assumptions underpin this message:

The bottleneck is in the compute path, not the server infrastructure: The assistant assumes that the GPU utilization problem can be diagnosed by tracing the forward pass code, rather than by profiling network, CPU scheduling, or Python overhead. This is a reasonable assumption given that the GPUs show 100% utilization but low power draw — a classic sign of memory-bound or synchronization-bound kernels.
The FP4 tensor cores are the right hardware to measure: The assistant assumes that the model's compute should be dominated by FP4 matrix multiplications (the MoE expert GEMMs), and that if these are running efficiently, the GPU power should be higher. This assumption is validated by the model architecture — GLM-5 is a MoE model where most FLOPs come from expert computation.
The subagent can trace the code effectively: The assistant assumes that the subagent has access to the codebase and can navigate it to find the relevant paths. Given that the subagent runs in the same environment, this is reasonable.
The allreduce communication is a suspect: While not explicitly stated in this message, the context (message 728) shows the assistant already suspecting that "allreduce communication — PCIe bottleneck" is a factor. This assumption shapes the investigation.

Potential Mistakes and Incorrect Assumptions

The most significant potential mistake in this message is what it doesn't say. The assistant frames the investigation purely around "what operations are happening per forward pass" — a static code analysis. But the low GPU utilization could also be caused by:

Dynamic scheduling issues: The sglang server's batching scheduler might not be feeding enough work to the GPU continuously. The assistant later discovers that max-running-requests was a critical tunable, but this message doesn't consider scheduling dynamics.
CPU-side bottlenecks: The Python overhead of the model runner, tokenization, or request handling could be limiting the rate at which work is dispatched to the GPU. This is a common issue in inference servers.
Memory bandwidth saturation: Even at 42% TDP, the GPU could be memory-bandwidth-bound if the kernels are not compute-intensive enough. The FP4 tensor cores might be stalled waiting for data from HBM.
The "100% utilization" mirage: The assistant accepts the user's framing that GPU utilization is 100% but power is low. However, different tools report utilization differently — nvidia-smi reports "GPU utilization" as the fraction of time at least one kernel was running, which can be 100% even if only a tiny fraction of SMs are active. The subagent's analysis (visible in the next message, 730) actually addresses many of these points, finding that the primary bottleneck is indeed the allreduce fusion being disabled on SM120. So the assumption that the compute path is the right place to look was validated. But the message itself doesn't hedge against the possibility that the bottleneck might be elsewhere. Another subtle issue: the assistant asks "how much of the GPU compute (FP4 tensor cores) are we actually using?" This presupposes that the FP4 tensor cores are the main compute resource. But for attention operations (which use BF16), the GPU uses different compute units. If attention is a larger fraction of the forward pass than expected, the FP4 tensor core utilization might not be the right metric.

Input Knowledge Required

To understand this message fully, one needs:

The GLM-5-NVFP4 model architecture: It's a Mixture-of-Experts transformer with FP4 quantized expert weights and BF16 attention. Understanding MoE routing, expert parallelism, and the split between attention and expert computation is essential.
The RTX PRO 6000 Blackwell (SM120) hardware: This consumer Blackwell variant has 100KB shared memory per SM (vs 228KB on datacenter SM100), 600W TDP, and FP4 tensor core support. The SM120 architecture is a cut-down version of the datacenter Blackwell.
The SGLang inference framework: SGLang is a serving system for large language models. Key concepts include the model runner, attention backends (flashinfer, triton, trtllm), MoE backends, and the allreduce communicator.
The concept of allreduce fusion: In tensor-parallel inference, allreduce operations synchronize gradients across GPUs. "Fusion" means overlapping these communication operations with computation to hide latency. Without fusion, the GPU stalls during PCIe transfers.
The NVFP4 quantization format: A 4-bit floating point format (E2M1) used for the expert weights. The tensor cores can directly compute on this format, which is why it's efficient.
The conversation history: The user's repeated concerns about power utilization, the research document about SM120 shared memory constraints, and the earlier debugging of NaN crashes and PCIe bottlenecks.

Output Knowledge Created

This message produces several important outputs:

A comprehensive forward pass analysis: The subagent traces the entire inference path, identifying data types at each stage, kernel launch patterns, and communication overhead. This becomes the basis for targeted fixes.
Identification of the allreduce fusion gap: The subagent discovers that communicator.py checks for SM90 or SM100 but not SM120, meaning allreduce fusion is disabled on the RTX PRO 6000. This is the root cause of the low GPU utilization.
A prioritized action plan: The next message (730) shows the assistant distilling the findings into a clear priority list: (1) fix allreduce fusion for SM120, (2) fix MoE runner backend auto-detection, (3) fix allreduce fusion auto-enable in server_args.py, (4) investigate custom allreduce for PCIe-only GPUs.
Confirmation that FP4 tensor cores are correctly used: The subagent verifies that the expert GEMMs are indeed using FP4 compute, ruling out the hypothesis that weights were being dequantized to BF16 before computation.
A deeper understanding of SM120 limitations: The analysis reveals that FlashInfer's allreduce fusion kernels use TRT-LLM communication primitives that only support SM90/SM100, and that patching them for SM120 requires non-trivial kernel work.

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message is compact but revealing. The phrase "Good, server is still running" shows a checkpoint mentality — the assistant is verifying stability before proceeding to deeper investigation. This is a hallmark of systematic debugging: ensure the system is in a known good state before perturbing it.

The shift from "what's the throughput?" to "what operations are happening?" represents a move from observational to mechanistic analysis. The assistant is essentially saying: "I've measured the symptoms (low power, moderate throughput), now I need to understand the mechanism."

The question "how much of the GPU compute (FP4 tensor cores) are we actually using?" is particularly insightful. It recognizes that GPU utilization metrics are aggregate and can hide the distinction between "the GPU is busy" and "the GPU is doing useful work." The FP4 tensor cores are the specialized hardware that makes NVFP4 quantization efficient — if they're idle, the whole point of using NVFP4 is undermined.

The decision to use a task subagent rather than manual code reading reflects an understanding of the codebase's complexity. The assistant could have tried to trace the forward pass by reading files sequentially, but that would be slow and error-prone. The subagent can systematically explore the code, following function calls across files, and return a synthesized analysis.

The prompt to the subagent is structured as a series of specific questions about data types, compute paths, and bottlenecks. This shows the assistant has a mental model of what the forward pass should look like and is testing specific hypotheses: "Are the attention weights being sent over PCIe in BF16 instead of FP4?" "Is the MoE compute using FP4 tensor cores?" "Is allreduce fusion enabled?"

Aftermath and Impact

The subagent's analysis, returned in the task result, was a turning point. It confirmed that allreduce fusion was disabled on SM120, meaning every allreduce operation (156 per forward pass: 2 per layer × ~78 layers) was serialized with compute. The GPU SMs were sitting idle during PCIe transfers, explaining the low power draw.

The next message (730) shows the assistant immediately acting on these findings, prioritizing fixes and beginning to patch the code. The subsequent messages (731-734) show the assistant reading the relevant source files and preparing patches.

This single message — a brief delegation to a subagent — thus set in motion the chain of discoveries that would ultimately lead to significant performance improvements. It exemplifies a key principle of debugging complex systems: when surface-level metrics don't make sense, trace the actual execution path. The GPU was reporting 100% utilization, but it was the wrong kind of busy — busy waiting for data over PCIe instead of computing on FP4 tensor cores.

Conclusion

Message 729 is a masterclass in diagnostic reasoning. Faced with a puzzling discrepancy between GPU utilization (100%) and GPU power draw (42% of TDP), the assistant resisted the temptation to keep tweaking parameters and instead asked a more fundamental question: what is the GPU actually computing? By delegating a systematic code trace to a subagent, the assistant uncovered the root cause — disabled allreduce fusion for SM120 — and set the stage for targeted fixes that would eventually push throughput toward 4,000 tok/s.

The message's power lies not in what it says, but in what it represents: the transition from black-box optimization to white-box understanding. In any complex system, the most valuable debugging step is often the simplest: stop guessing and start tracing.