The Turning Point: Diagnosing GPU Underutilization in GLM-5-NVFP4 Inference on SM120 Blackwell

Introduction

In the high-stakes world of large language model inference, every watt of GPU power counts. When deploying the massive GLM-5-NVFP4 mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant in this opencode session encountered a perplexing problem: despite achieving respectable token throughput, the GPUs were barely breaking a sweat. Power draw hovered around 250W per GPU — a mere 42% of the 600W thermal design power (TDP) ceiling. Something was starving the hardware, and the answer lay deep within the SGLang inference stack.

Message 730 ([msg 730]) marks the critical inflection point in this debugging journey. It is the moment when the assistant, having just received the results of a comprehensive subagent analysis of the GLM-5 forward pass, synthesizes the findings into a coherent root-cause diagnosis and pivots from investigation to active remediation. This message is a masterclass in structured debugging under uncertainty — it demonstrates how to take a vague symptom ("GPUs not fully fed"), decompose it into testable hypotheses, commission deep research via a subagent task, and then translate those findings into a prioritized action plan with concrete code changes.

The Message in Full

Let us examine the message exactly as written:

Excellent research. This is very revealing. Let me summarize the key findings and start fixing them:

>

## Root Cause: GPUs Only at 42% TDP

>

The main bottleneck is PCIe allreduce stalls without compute overlap:

>

1. CRITICAL: FlashInfer allreduce fusion is disabled on SM120 — communicator.py:78 checks for SM90 or SM100 only, not SM120. This means every allreduce (2 per layer × ~78 layers = 156 allreduces per forward pass) is serialized with compute — GPU SMs sit idle during PCIe transfers.

>

2. SIGNIFICANT: Custom allreduce disabled for >2 PCIe-only GPUs — falls through to NCCL with higher launch overhead.

>

3. MODERATE: MoE runner backend not auto-selected for SM120 in server_args.py.

>

Let me fix the critical issue first — enabling allreduce fusion for SM120:

The message then shows a todowrite block with four prioritized tasks, the first marked "in_progress": fixing FlashInfer allreduce fusion for SM120 in communicator.py.

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, we must trace the chain of reasoning that led to it. The story begins three messages earlier, at [msg 723], when the user pointedly observed that GPU power numbers were stuck around 330W — not the expected 600W TDP — and asked the assistant to debug what was starving the GPUs. The user also referenced a research document about SM120 attention kernel shared memory fixes, hinting that the RTX PRO 6000 (compute capability SM120) is architecturally distinct from datacenter Blackwell GPUs (SM100) in ways that the inference stack might not be handling correctly.

The assistant's initial response at [msg 724] reveals a critical assumption: it believed the TDP was 300W, not 600W. This assumption was based on the RTX PRO 6000's consumer-grade branding — the assistant conflated it with lower-power variants. At 300W TDP, 250W draw would be a respectable 83% utilization. The assistant nearly dismissed the concern. But the user persisted, and the assistant dutifully checked at [msg 727], discovering the truth: the RTX PRO 6000 Server Edition has a 600W power limit. The gap was not 17% but 58% — a massive, unexplained deficit.

This discovery reframed the entire problem. The assistant now had a clear quantitative target: find out why the GPUs were only consuming 250W when they could consume 600W. It began profiling the GPU state — checking clocks, memory bandwidth, and power readings — and then dispatched a subagent task at [msg 729] with the prompt: "Analyze GLM-5 forward pass compute." This subagent was tasked with tracing every operation in the forward pass, identifying data types, and pinpointing where the GPU was stalling.

The subject message ([msg 730]) is the assistant's synthesis of that subagent's report. It represents the moment when raw data crystallizes into actionable insight.

How Decisions Were Made

The message reveals a clear decision-making framework. The assistant organizes its findings into three categories ranked by severity: CRITICAL, SIGNIFICANT, and MODERATE. This prioritization is not arbitrary — it reflects the estimated impact on GPU utilization.

Decision 1: FlashInfer allreduce fusion is the primary bottleneck. The reasoning is quantitative: the GLM-5 model has approximately 78 transformer layers, and each layer performs two allreduce operations (one for attention, one for MoE). That is 156 allreduces per forward pass. If each allreduce requires a PCIe round-trip without overlapping with compute, the GPU SMs spend a substantial fraction of time idle, waiting for data to arrive from peer GPUs. The assistant identifies that communicator.py line 78 gates allreduce fusion on is_sm90_supported or is_sm100_supported, explicitly excluding SM120. This is the smoking gun.

Decision 2: Fix the fusion gate first. The assistant commits to modifying communicator.py to add SM120 support for allreduce fusion. This is the highest-leverage change because it directly addresses the primary bottleneck. The message shows the assistant already transitioning to implementation mode — the first todo is marked "in_progress."

Decision 3: Investigate custom allreduce for PCIe-only GPUs. The assistant notes that the custom allreduce implementation is disabled for configurations with more than two PCIe-only GPUs. With eight GPUs communicating exclusively over PCIe (no NVLink), the system falls through to NCCL, which has higher launch overhead. This is marked as "pending" — important but secondary to the fusion fix.

Decision 4: Fix SM120 auto-detection in server_args.py. The MoE runner backend is not automatically selected for SM120, which may cause suboptimal kernel selection. This is marked as "pending" with lower priority.

Assumptions Made by the Assistant

The message rests on several assumptions, most of which are reasonable but worth examining:

  1. The allreduce fusion fix will significantly increase GPU utilization. The assistant assumes that enabling allreduce fusion — overlapping communication with computation — will keep GPU SMs busy during PCIe transfers. This is a well-established optimization in distributed training and inference, but the actual speedup depends on whether the fused operations can fully occupy the SMs while waiting for data.
  2. The number of allreduces is exactly 2 per layer × ~78 layers. The "~78" qualifier suggests uncertainty about the exact layer count. The assistant is approximating based on the GLM-5 architecture. If the actual count differs, the estimated impact could shift.
  3. PCIe bandwidth is the binding constraint. The assistant implicitly assumes that the GPUs are not compute-bound — that if allreduce stalls were eliminated, the FP4 tensor cores would have enough work to reach 600W. This assumption is supported by the observation that GPU power is low even at high concurrency, suggesting the compute units are underfed.
  4. The SM120 architecture can support allreduce fusion with the same primitives as SM90/SM100. This assumption proved fragile — later in the session (as described in the chunk summary), attempts to patch allreduce fusion for SM120 caused performance to drop to 236 tok/s and power to 125W, suggesting synchronization issues. The assumption that "just add SM120 to the gate" would work turned out to be incorrect, revealing deeper architectural differences in the TRT-LLM communication kernels.
  5. The MoE runner backend auto-detection is a moderate issue. The assistant assumes that fixing the backend selection will yield measurable but not transformative gains. This is a reasonable heuristic given that the model is already running and generating tokens.

Mistakes and Incorrect Assumptions

The most notable mistake in this message is not in what it says, but in what it doesn't yet know. The assistant assumes that enabling allreduce fusion for SM120 is a straightforward patch — simply add SM120 to the architecture check in communicator.py. However, as the subsequent chunk summary reveals, this assumption proved incorrect:

"Attempts to patch the flashinfer kernel to add SM120 support (by modifying the architecture gate and version check) allowed the server to start, but the fusion performed poorly, dropping throughput to 236 tok/s and power to 125W, suggesting synchronization issues on SM120."

The root cause is that FlashInfer's allreduce fusion relies on TRT-LLM communication kernels that were specifically designed for SM90 (Hopper) and SM100 (datacenter Blackwell). These kernels use architecture-specific synchronization primitives and shared memory configurations that do not exist or behave differently on SM120. Simply opening the gate allowed the code path to execute, but the underlying kernels were not validated for the SM120 memory subsystem, leading to silent correctness issues or severe performance degradation.

This is a classic pitfall in GPU programming: architecture compatibility gates exist for a reason. The SM120 architecture, despite sharing the "Blackwell" branding with SM100, has fundamentally different characteristics — smaller shared memory (100KB vs 228KB), different warp scheduling, and potentially different tensor core capabilities. The assistant's initial assumption that the gate was an oversight rather than a deliberate exclusion was optimistic.

Another subtle issue: the assistant describes the power gap as "42% TDP" (250W out of 600W), but this conflates instantaneous power draw during inference with the thermal design power ceiling. GPUs rarely sustain full TDP during inference workloads, which tend to be memory-bandwidth-bound rather than compute-bound. A more nuanced analysis would compare power draw during inference to power draw during a compute-bound workload (e.g., a GEMM microbenchmark) to establish the true ceiling. The assistant implicitly assumes that 600W is achievable for this workload, which may not be true — the FP4 tensor cores may simply not draw enough current to hit the power limit even when fully utilized.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several domains:

SGLang architecture: The message references communicator.py, server_args.py, and concepts like "allreduce fusion," "MoE runner backend," and "FlashInfer." The reader must understand that SGLang is a serving system for large language models, that it uses tensor parallelism across multiple GPUs, and that allreduce is the communication primitive for synchronizing gradients/hidden states across GPUs.

GPU architecture and CUDA compute capabilities: The message distinguishes between SM90 (Hopper/H100), SM100 (datacenter Blackwell/B200), and SM120 (consumer Blackwell/RTX PRO 6000). The reader must understand that these are NVIDIA's streaming multiprocessor architecture versions, each with different capabilities, shared memory sizes, and supported features.

Mixture-of-Experts (MoE) model structure: The GLM-5 model uses MoE layers, which have multiple "expert" sub-networks. The allreduce pattern (2 per layer) reflects the need to synchronize both the attention output and the MoE output across GPUs.

PCIe communication topology: The message assumes the reader understands that in a multi-GPU system without NVLink, GPUs communicate over PCIe, which has higher latency and lower bandwidth than NVLink. The "custom allreduce" optimization attempts to mitigate this.

The session's prior context: The reader needs to know that the team has been battling NaN crashes, attention backend selection, and PCIe P2P limitations throughout the earlier segments of this session. The SM120 attention fix research document (referenced in [msg 723]) provides critical background on why SM120 is different from SM100.

Output Knowledge Created

This message creates several important outputs:

  1. A prioritized root-cause analysis: The assistant distills a complex subagent research report into three clear, ranked bottlenecks. This prioritization enables efficient resource allocation — fix the biggest problem first.
  2. A concrete action plan with code locations: The message identifies specific files (communicator.py, server_args.py) and specific line numbers (78) where changes are needed. This transforms abstract performance analysis into actionable engineering work.
  3. A quantitative model of the bottleneck: By calculating "2 allreduces per layer × ~78 layers = 156 allreduces per forward pass," the assistant provides a tangible sense of the problem's scale. This number can be used to estimate the potential speedup from fusion.
  4. A decision record: The message documents why the assistant chose to fix allreduce fusion first, rather than the MoE backend or custom allreduce. This reasoning is valuable for future debugging and for understanding the session's trajectory.
  5. A transition from analysis to action: The todowrite block with one item "in_progress" signals that the assistant has finished investigating and is now actively modifying code. This is a clear state transition that structures the subsequent conversation.

The Thinking Process Visible in the Message

The message reveals a structured, analytical thinking process. The assistant begins with a summary statement — "Excellent research. This is very revealing." — which serves as both acknowledgment and transition. It then presents the root cause as a single, clear sentence: "PCIe allreduce stalls without compute overlap."

The three bullet points are ordered by estimated impact, not by ease of fix. The CRITICAL item (allreduce fusion) is the hardest to fix (requires understanding the fusion kernel and SM120 compatibility) but has the highest potential payoff. The MODERATE item (MoE backend auto-detection) is likely the easiest to fix but has lower impact. This ordering demonstrates strategic thinking: fix the bottleneck that matters most, even if it is harder.

The use of "~78 layers" (with the tilde) is a subtle but important signal of intellectual honesty. The assistant is not certain about the exact layer count, so it qualifies the number. This prevents the reader from treating the 156-allreduce figure as precise when it is an approximation.

The message also shows the assistant managing its own attention through the todowrite system. By explicitly writing down four tasks and marking one as "in_progress," the assistant externalizes its working memory. This is a metacognitive strategy — by committing tasks to text, the assistant reduces cognitive load and ensures it will not forget secondary issues while focusing on the primary fix.

Conclusion

Message 730 is a pivotal moment in a complex debugging session. It represents the transition from investigation to intervention — the point at which the assistant has gathered enough evidence to form a hypothesis and is ready to test it through code changes. The message is notable for its clarity of reasoning, its prioritization of fixes by impact, and its willingness to commit to a specific diagnosis even in the face of uncertainty.

The subsequent events — where the allreduce fusion patch caused performance to collapse — do not invalidate the reasoning in this message. Rather, they highlight the gap between architectural compatibility (the gate check) and architectural correctness (the kernel implementation). The assistant correctly identified the gate as a problem, but underestimated the depth of the underlying kernel differences between SM100 and SM120. This is a valuable lesson in GPU software engineering: architecture gates in open-source projects are often the visible tip of a much larger iceberg of assumptions about shared memory sizes, register counts, synchronization primitives, and tensor core capabilities.

In the end, the assistant's willingness to dig into the code, commission a subagent analysis, and act on the findings — even when the first attempt failed — exemplifies the kind of systematic debugging that high-performance computing demands. Message 730 captures that spirit at its most optimistic moment: the moment before the patch, when the solution seemed within reach.