The Power Draw Revelation: Diagnosing GPU Underutilization in Blackwell MoE Inference

Introduction

In the high-stakes world of large-scale ML inference optimization, few moments are as telling as the one captured in message 717 of this opencode session. After a grueling sequence of debugging NaN crashes, patching kernel code, tuning server parameters, and ultimately achieving a stunning 3,740 tok/s throughput for the GLM-5-NVFP4 model on 8× RTX PRO 6000 Blackwell GPUs, the assistant pauses to ask a seemingly simple question: how hard are the GPUs actually working? The answer, delivered by nvidia-smi, reveals a paradox that would define the next phase of optimization: 100% GPU utilization, yet power draw hovering at a mere ~250W per GPU — barely 42% of the 600W thermal design power (TDP) these Blackwell monsters are rated for.

This message is not about achieving a new throughput record. It is about the moment of diagnostic clarity that follows success — the realization that the hardware is being compute-utilized but not power-utilized, and that the bottleneck has shifted from software configuration to fundamental system architecture. It is a masterclass in how to read the silent signals that GPUs emit when they are waiting on something other than computation.

The Message in Full

The assistant issues two parallel bash commands via SSH to the LXC container running on the Proxmox host:

[assistant] ~87W idle. Now let me start a benchmark and check power during it:
[bash] ssh root@10.1.230.174 "source /root/ml-env/bin/activate && python3 -m sglang.bench_serving --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --base-url http://localhost:8000 --dataset-name random --num-prompts 1024 --random-input-len 256 --random-output-len 128 --request-rate inf 2>&1 | tail -25" &
[bash] sleep 20 && ssh root@10.1.230.174 "nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv,noheader"
0, 100 %, 256.54 W
1, 100 %, 248.95 W
2, 100 %, 255.94 W
3, 100 %, 247.21 W
4, 100 %, 248.81 W
5, 100 %, 249.72 W
6, 100 %, 255.02 W
7, 100 %, 251.61 W

The first command launches the same 1024-concurrency benchmark that previously yielded 3,740 tok/s, but this time it is backgrounded (&) so that the assistant can concurrently probe GPU state. The second command waits 20 seconds — enough time for the benchmark to reach steady-state saturation — then queries all 8 GPUs for utilization percentage and power draw. The results are uniform across all GPUs: 100% utilization, but power figures between 247W and 257W.

Why This Message Was Written: The Reasoning and Motivation

The assistant's motivation is rooted in a fundamental principle of performance engineering: throughput numbers alone cannot tell you where the bottleneck is. The previous messages (714–715) had established that the system could achieve 3,740 tok/s at 1024 concurrency, with a peak output rate of 3,945 tok/s. These are impressive numbers — a roughly 4.25× improvement over the ~880 tok/s baseline from earlier in the session. But any experienced performance engineer knows that the first question after a big win is always: what is still limiting us?

The assistant had already identified one major bottleneck earlier in the session: FlashInfer's allreduce fusion was disabled on SM120 (the Blackwell consumer GPU architecture) because the underlying TRT-LLM communication kernels only supported SM90 (Hopper) and SM100 (datacenter Blackwell). This meant that the 8 GPUs were communicating via standard NCCL allreduce over PCIe, rather than using the fused kernel that would dramatically reduce communication overhead. The assistant had attempted to patch FlashInfer to add SM120 support, but the result was catastrophic — throughput dropped to 236 tok/s and power to 125W — suggesting synchronization primitives that didn't work correctly on the consumer architecture.

Given this context, the assistant's decision to check GPU power draw during a benchmark is a logical diagnostic step. The reasoning chain is:

  1. We have high throughput, but is the hardware saturated? The 3,740 tok/s number is impressive, but without knowing GPU utilization, it's impossible to tell if we're hitting a compute wall, a memory wall, a communication wall, or a software scheduling wall.
  2. The allreduce fusion patch failed, so communication is likely the bottleneck. If the GPUs spend most of their time waiting for each other to finish allreduce operations, they would show high utilization (because the GPU is busy during the compute portion) but low power (because waiting on PCIe does not consume much power).
  3. Power draw is a better indicator of actual work than utilization percentage. GPU utilization reported by nvidia-smi measures what fraction of time the GPU has at least one kernel running. But a GPU can be "100% utilized" while running lightweight kernels that don't stress the memory system or the compute units. Power draw, by contrast, directly reflects the physical work being done — high power means high memory bandwidth usage, high ALU activity, or both.
  4. If power is low despite 100% utilization, the GPU is stalled on something external. This is the key insight. A GPU that is truly compute-bound on matrix multiplications would draw close to its TDP. A GPU that is spending most of its time waiting for data from other GPUs (via PCIe allreduce) would show high utilization (because it's constantly launching small kernels) but low power (because those kernels are not keeping the compute units fed). The assistant's hypothesis, unstated but clearly implied by the action, is: the GPUs are allreduce-bound, and the power draw will confirm this.

The Diagnostic Result: A Paradox Revealed

The results confirm the hypothesis, but with an important nuance. All 8 GPUs report exactly 100% utilization — they are never idle. Yet their power draw clusters tightly around 250W, compared to a 600W TDP. This is a textbook sign of communication-bound execution in a multi-GPU setting.

To understand why, consider the anatomy of a single decoding step in a mixture-of-experts (MoE) model like GLM-5-NVFP4 running with tensor parallelism 8:

  1. Each GPU receives its shard of the input tokens.
  2. Each GPU computes its portion of the attention mechanism (using the NSA decode backend).
  3. Each GPU computes its portion of the MoE feedforward layers.
  4. All GPUs must allreduce their outputs to produce the final result for each token.
  5. The allreduce operation requires each GPU to send its partial sums to all other GPUs, then receive and accumulate the results. Steps 1–3 are compute-heavy and would normally draw significant power. Step 4 is communication-heavy and involves relatively little computation — just waiting for data to arrive over PCIe. If the allreduce is the bottleneck, the GPUs will spend a significant fraction of their time in step 4, during which they are "utilized" (they have a kernel running — the NCCL allreduce kernel) but drawing minimal power (because the kernel is mostly waiting for PCIe transactions to complete). The 250W reading tells us that the GPUs are spending roughly 42% of their time doing actual compute work (which would draw ~600W) and 58% of their time waiting on communication (which draws ~87W idle power, plus some overhead). This is consistent with a system where PCIe bandwidth is the primary limiter.

Assumptions Made by the Assistant

The assistant makes several assumptions in this message, most of which are well-justified:

  1. The benchmark will reach steady state within 20 seconds. This is a reasonable assumption given that the same benchmark with 1024 prompts and 256 input tokens + 128 output tokens typically runs for several minutes. Waiting 20 seconds ensures the system has moved past the initial prefill phase and is in the decode-heavy steady state.
  2. Power draw is a meaningful proxy for compute intensity. This is a well-established principle in GPU performance analysis. NVIDIA GPUs have sophisticated power management that adjusts voltage and frequency based on workload. A GPU drawing near its TDP is doing sustained heavy computation; a GPU drawing far below its TDP is either idle or stalled.
  3. The nvidia-smi query is non-invasive. Running nvidia-smi during a benchmark does not significantly perturb the GPU state. The query is handled by the NVIDIA driver's management infrastructure and has negligible overhead.
  4. All 8 GPUs should show similar behavior. In a tensor-parallel setup with homogeneous hardware, all GPUs execute the same workload (different data shards) and should exhibit similar power draw. The tight clustering of values (247–257W) confirms this assumption.
  5. The idle power baseline (~87W) is correct. The assistant had just checked idle power before launching the benchmark, establishing a baseline of ~87W across all GPUs. This is consistent with the known idle power consumption of RTX PRO 6000 Blackwell GPUs.

Mistakes and Incorrect Assumptions

While the message itself is sound, there are subtle points worth examining:

  1. GPU utilization at 100% can be misleading. The assistant implicitly trusts the utilization.gpu metric from nvidia-smi, but this metric has a well-known limitation: it reports the fraction of time that any kernel is executing, not the fraction of compute capacity being used. A GPU running a lightweight NCCL allreduce kernel that barely touches the CUDA cores will still show 100% utilization if that kernel runs continuously. The power reading is the more informative metric, and the assistant correctly prioritizes it.
  2. The 20-second delay might not capture the most communication-heavy phase. In MoE inference, the ratio of compute to communication varies depending on batch size. At very high concurrency (1024 requests), the batch size per GPU is large, which increases the compute-to-communication ratio (because matrix multiplications scale with batch size, while allreduce overhead scales with model dimension, which is fixed). It's possible that the power draw would be even lower at smaller batch sizes, where the allreduce overhead is more dominant. The assistant does not test this, but the 1024-concurrency case is the most relevant for the throughput-maximization goal.
  3. The assistant does not check memory clock or memory utilization. Power draw alone cannot distinguish between compute-bound and memory-bound scenarios. A GPU that is memory-bandwidth-bound (e.g., constantly reading from HBM) would also draw significant power, but the assistant cannot rule out this possibility without additional metrics like utilization.memory or memory clock frequency. However, given the known allreduce limitation on SM120, the communication-bound hypothesis is the most plausible.

Input Knowledge Required to Understand This Message

To fully appreciate the significance of this message, the reader needs:

  1. Understanding of tensor parallelism (TP). The model is split across 8 GPUs, with each GPU holding a shard of the weights. Every decoding step requires an allreduce operation to synchronize the partial results. The overhead of this allreduce grows with the number of GPUs and is limited by the interconnect bandwidth.
  2. Knowledge of NVIDIA GPU power characteristics. The RTX PRO 6000 Blackwell GPU has a 600W TDP, meaning it can draw up to 600W under sustained heavy compute load. Consumer and pro-viz GPUs typically have higher TDPs than datacenter GPUs (e.g., H100 at 700W, B200 at 700W) but are optimized for different workloads.
  3. Familiarity with the PCIe topology of the test system. Earlier in the session (Segment 3), the assistant discovered that each GPU is on its own PCIe root complex, meaning P2P (peer-to-peer) DMA between GPUs is impossible. All inter-GPU communication must go through the CPU's PCIe controller and system memory, adding latency and reducing bandwidth compared to direct GPU-to-GPU NVLink connections.
  4. Context about the FlashInfer allreduce fusion patch attempt. The assistant had previously tried to enable FlashInfer's fused allreduce kernel on SM120 by modifying the architecture gate in the FlashInfer source code. This attempt failed, producing worse throughput than the unfused baseline. The current message is a direct consequence of that failure — the assistant is now quantifying the cost of running without fused allreduce.
  5. Understanding of the relationship between GPU power draw and workload type. Compute-heavy workloads (matrix multiplications, convolutions) draw high power because they keep the CUDA cores and memory system busy. Communication-heavy workloads (allreduce, data transfer) draw lower power because the GPU is mostly waiting for data to arrive over the PCIe bus.

Output Knowledge Created by This Message

This message produces several pieces of actionable knowledge:

  1. A quantified baseline for GPU power under load: ~250W per GPU at 1024 concurrency, with 100% utilization. This is a reference point for future optimization attempts — any change that increases power draw toward 600W while maintaining or improving throughput would indicate better compute utilization.
  2. Confirmation that the allreduce bottleneck is real and significant. The 250W reading, compared to the 600W TDP, implies that the GPUs are spending more than half their time waiting on communication. This quantifies the opportunity cost of not having fused allreduce on SM120.
  3. Evidence that PCIe bandwidth, not compute capacity, is the primary limiter. In a system with NVLink or NVSwitch, the allreduce would be much faster, and the GPUs would spend more time computing and less time communicating. The power draw would be closer to the TDP.
  4. A diagnostic methodology for future optimization work. The assistant demonstrates a pattern: establish idle baseline, launch workload, wait for steady state, probe power and utilization. This is a reusable technique that can be applied to any configuration change.
  5. Implicit validation of the LXC container setup. The fact that all 8 GPUs show uniform behavior (100% utilization, ~250W power) confirms that the LXC container is correctly passing through the GPUs and that the NVIDIA driver is functioning properly. This is non-trivial — earlier in the session (Segment 4–5), the assistant struggled with CUDA initialization failures in the LXC container due to HMM (Heterogeneous Memory Management) issues in the nvidia_uvm module.

The Thinking Process Visible in This Message

The assistant's reasoning is visible in the structure of the message itself. The opening line — "~87W idle. Now let me start a benchmark and check power during it" — reveals the thought process:

  1. The assistant has just checked idle power (in the previous message, msg 716) and seen ~87W. This establishes a baseline.
  2. The assistant hypothesizes that power draw under load will be significantly lower than TDP. If the assistant expected near-TDP power draw, there would be no need to check — the GPUs would be fully saturated. The very act of checking implies doubt about whether the hardware is being fully utilized.
  3. The assistant designs a concurrent experiment. By launching the benchmark in the background and then probing power after a delay, the assistant avoids the need to run two separate tests. This is efficient and eliminates the risk of cold-start effects from a fresh server launch.
  4. The assistant chooses to query all 8 GPUs individually. The --query-gpu=index,utilization.gpu,power.draw format with --format=csv,noheader produces a compact, machine-readable output that shows per-GPU values. The assistant could have queried a single GPU, but checking all 8 reveals whether the workload is balanced — which it is, with all GPUs within a narrow 10W range.
  5. The assistant does not immediately act on the result. The message ends with the power data and no further commentary. This is a "data gathering" message — the assistant will use this information in subsequent messages to decide on the next optimization direction. The lack of immediate action suggests the assistant is processing the implications: if the GPUs are communication-bound, the solution is not more compute optimization but either (a) reducing communication overhead, (b) increasing effective bandwidth, or (c) changing the parallelism strategy (e.g., switching from TP8 to TP4+PP2 to reduce allreduce size).

The Broader Context: A Session of Breakthroughs and Barriers

This message sits at a pivotal moment in the optimization journey. The session began with the GLM-5-NVFP4 model barely managing 880 tok/s, plagued by NaN crashes during decode that required careful selection of attention backends (the trtllm NSA backend for decode, flashinfer for prefill). The assistant then unlocked a series of improvements: enabling FlashInfer CUTLASS MoE autotune for SM120 by patching model_runner.py, raising --max-running-requests from 64 to 1024, disabling CUDA graphs and radix cache to avoid a page_table_1_flattened bug in the NSA attention path. Each change compounded to produce the 4.25× throughput improvement.

But the allreduce fusion barrier remained. The assistant's attempt to patch FlashInfer to support SM120 had failed spectacularly, and the NCCL tuning and --num-continuous-decode-steps experiments had yielded no gains. The power draw data in this message crystallizes the problem: the GPUs are compute-ready but communication-starved. The Blackwell architecture's SM120 variant lacks the fused allreduce kernels that make multi-GPU inference efficient on datacounter Blackwell (SM100) and Hopper (SM90).

This is not a problem that can be solved by software configuration alone. It requires either (a) upstream support from FlashInfer/TRT-LLM for SM120 allreduce fusion, (b) a hardware topology change (e.g., moving to a system with NVLink or better PCIe topology), or (c) a fundamental change in parallelism strategy. The assistant's subsequent exploration of expert parallelism (EP) and TP4+PP2 in later messages is a direct consequence of the insight gained here.

Conclusion

Message 717 is a masterclass in diagnostic humility. After achieving a 4.25× throughput improvement, the assistant does not declare victory. Instead, it asks a deeper question: are we actually using the hardware we paid for? The answer — 250W out of 600W TDP — reveals that the GPUs are spending more than half their time waiting for each other. This is the kind of insight that separates superficial optimization from deep system understanding.

The message also demonstrates the power of combining multiple data sources. Throughput numbers alone would have suggested a well-optimized system. GPU utilization numbers alone would have suggested a fully saturated system. But the combination of high utilization with low power draw tells a more nuanced story: the GPUs are busy, but they are not working hard. They are waiting.

For anyone engaged in multi-GPU inference optimization, this message is a reminder that the most important diagnostic tool is not a profiler or a trace — it is the simple act of asking the hardware what it is experiencing. The GPUs cannot speak, but they can signal through power draw, utilization, and temperature. Learning to read these signals is what separates the merely competent from the truly effective.