The Power Puzzle: Diagnosing GPU Utilization in GLM-5-NVFP4 Inference

Introduction

In the middle of an intensive performance optimization session for the GLM-5-NVFP4 large language model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message captures a moment of diagnostic insight. Message 939 in the conversation is brief—a single observation followed by a benchmark command and its results—but it represents a critical pivot in the assistant's reasoning about where the true bottleneck lies.

The message reads:

[assistant] Still ~235W per GPU. Let me now try a longer output test to get sustained decode throughput: [bash] ssh root@10.1.230.174 'source /root/ml-env/bin/activate && echo "=== 1024 reqs, 256 output ===" && python3 -m sglang.bench_serving \ --backend sglang --host 127.0.0.1 --port 8000 \ --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 \ --dataset-name random --random-input-len 128 --random-output-len 256 \ --num-prompts 1024 --request-rate 999 \ 2>&1 | grep -E "Output token throughput|Total token throughput|Peak output|Mean TPOT|Concurrency:"' === 1024 reqs, 256 output === Output token throughput (tok/s): 1876.30 Peak output token throughput (tok/s): 4263.00 Total token throughput (tok/s): 2803.47 Concurrency: 633.43 Mean TPOT (ms): 394.78

To understand why this message matters, we must trace the reasoning thread that led to it—a thread that weaves together GPU architecture constraints, kernel efficiency analysis, and the relentless pursuit of throughput.

The Power Utilization Puzzle

The message opens with an observation: "Still ~235W per GPU." This single phrase carries immense weight. The RTX PRO 6000 Blackwell GPUs have a 600W thermal design power (TDP), yet during inference they are drawing only about 39% of that budget. For a performance engineer, this is both a clue and a provocation.

The assistant had just completed a benchmark at 2048 concurrency that showed a dramatic 28% throughput improvement—from 1,640 to 2,095 output tokens per second—achieved by tuning --max-running-requests to 2048 and --num-continuous-decode-steps to 8. Yet the GPU power draw remained stubbornly low. This suggested one of two possibilities: either the GPUs were fundamentally unable to utilize more power because the workload was compute-bound in a way that left functional units idle, or the benchmark workload itself was too short to reach steady-state operation where power would ramp up.

The assistant's earlier deep investigation into FP4 GEMM kernel efficiency on SM120 had already revealed a sobering picture. The CUTLASS kernels, which handle the matrix multiplications at the heart of the model's Mixture-of-Experts layers, achieve approximately 1,300 TFLOPS (70% of dense peak) only for very large matrices. During actual decode, per-expert batch sizes of roughly 16–64 tokens achieve merely 0.8 to 55 TFLOPS—between 0.02% and 3% of peak. The 99KB shared memory limit on SM120 prevents using larger CUTLASS tile configurations, creating a fundamental ceiling on kernel efficiency.

But the power question remained unresolved: was the low power draw a measurement artifact of short benchmarks, or a genuine reflection of architectural inefficiency?## The Reasoning Behind the Longer Output Test

The assistant's decision to run a benchmark with 256 output tokens (double the previous 128) was a deliberate diagnostic maneuver. The hypothesis was straightforward: if the GPUs were not being fully exercised because the decode phase was too short, extending the output length would increase the sustained decode portion of the workload, potentially pushing power draw higher and revealing a different throughput characteristic.

This reasoning reflects a sophisticated understanding of how transformer inference works. In the prefill phase, the model processes the entire input prompt in parallel, achieving high arithmetic intensity and good GPU utilization. But during decode, tokens are generated one at a time, and the model must attend to the growing sequence of cached key-value pairs. The decode phase is where the MoE layers—with their 256 experts and 8 active experts per token—create the most challenging computational pattern: small, irregular matrix multiplications that struggle to saturate the GPU's tensor cores.

By doubling the output length from 128 to 256 tokens, the assistant was effectively asking: "If we force the model to spend more time in the decode regime, does throughput scale linearly, or do we hit a different bottleneck?" The answer would distinguish between a system that is compute-bound (where longer outputs should yield proportionally more tokens per second) and one that is memory-bound or communication-bound (where throughput might plateau or even degrade).

What the Results Revealed

The benchmark results were revealing. With 1,024 requests, 128 input tokens, and 256 output tokens, the system achieved:

• Output token throughput: 1,876.30 tok/s
• Peak output token throughput: 4,263.00 tok/s
• Total token throughput: 2,803.47 tok/s
• Concurrency: 633.43
• Mean TPOT (time per output token): 394.78 ms Comparing these numbers to the previous benchmark at 1,024 concurrency with 128 output tokens (which achieved 1,631 output tok/s, 3,250 total tok/s, and a mean TPOT of roughly 200 ms) reveals a nuanced picture. The output token throughput actually increased from 1,631 to 1,876 tok/s—a 15% improvement—despite the concurrency being lower (633 vs. ~1,024). However, the total token throughput dropped from 3,250 to 2,803 tok/s, and the mean TPOT doubled from approximately 200 ms to 395 ms. This is exactly what one would expect from a compute-bound system under longer decode sequences. The per-token latency increases because each decode step must attend to a longer KV cache, and the throughput in terms of output tokens per second actually improves because the system spends more time in the steady-state decode regime where the scheduler can batch requests efficiently. The concurrency of 633 (rather than the full 1,024) suggests that requests completed faster than new ones could be fed in, indicating that the system was not fully saturated.

Assumptions and Their Validity

The assistant made several implicit assumptions in this message. First, it assumed that extending output length would provide a clearer signal about sustained decode performance. This was a reasonable assumption, as longer decode sequences do reduce the relative contribution of prefill and allow the system to reach a more stable operating point.

Second, the assistant assumed that the GPU power draw of ~235W was potentially a measurement artifact of short benchmarks. This assumption was partially validated: the longer benchmark did show improved output throughput, suggesting that the decode phase was indeed not being fully exercised at shorter output lengths. However, the power draw during the longer benchmark was not explicitly measured, leaving the question partially open.

Third, the assistant assumed that the benchmark at 1,024 requests with 256 output tokens would be comparable to the earlier benchmark at 1,024 requests with 128 output tokens. This comparison is valid for understanding the scaling behavior, but the change in output length introduces a confounding variable: the KV cache size grows with output length, which affects memory pressure and can trigger different scheduling behavior.## The Broader Context: A Journey of Optimization

This message sits at a pivotal moment in a much larger optimization narrative. The assistant had already made significant progress:

1. Resolved the NaN decode crash by selecting working NSA backends (trtllm), enabling the model to run at all.
2. Established baseline throughput and identified virtualization-induced PCIe P2P latency as a key bottleneck.
3. Bypassed the virtualization bottleneck by moving to an LXC container on the Proxmox host, achieving bare-metal GPU topology with proper P2P access.
4. Enabled FlashInfer CUTLASS MoE autotune for SM120, which dramatically improved throughput from ~880 to ~3,740 tok/s.
5. Confirmed the model was compute-bound via TP4+PP2 benchmarking, ruling out allreduce latency as the primary bottleneck.
6. Deeply analyzed FP4 GEMM kernel efficiency, discovering that the GPUs draw only ~235W out of 600W TDP and that CUTLASS kernels achieve only 0.02–3% of peak during actual decode.
7. Achieved a 28% throughput improvement by tuning --max-running-requests to 2048 and --num-continuous-decode-steps to 8. Message 939 represents the next logical step in this chain: after achieving the 28% improvement, the assistant needed to understand whether the remaining headroom could be unlocked by changing workload characteristics or whether fundamental architectural constraints were the limiting factor.

The Thinking Process Visible in the Message

The message reveals a clear diagnostic thought process. The assistant begins with an observation ("Still ~235W per GPU"), formulates a hypothesis ("maybe the benchmarks are too short to reach steady-state decode"), designs an experiment ("let me try a longer output test"), executes it, and interprets the results.

This pattern—observe, hypothesize, experiment, analyze—is the hallmark of systematic performance engineering. The assistant is not randomly tweaking parameters; it is building a mental model of the system's behavior and testing specific predictions derived from that model.

The choice of 256 output tokens (rather than 128 or 512) is itself informative. Doubling the output length is a conservative change that should amplify decode-phase behavior without fundamentally changing the memory footprint or scheduling dynamics. A more aggressive change (e.g., 512 or 1024 output tokens) might have triggered different bottlenecks related to KV cache capacity or attention computation cost.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

• Transformer inference architecture: The distinction between prefill and decode phases, and how MoE layers create irregular compute patterns.
• GPU architecture: The concept of TDP and power draw as indicators of utilization, the SM120 architecture's 99KB shared memory limit, and how tensor core utilization varies with matrix dimensions.
• SGLang server parameters: What --max-running-requests and --num-continuous-decode-steps control, and how they affect batching behavior.
• Benchmarking methodology: How sglang.bench_serving works, what the metrics mean (output token throughput, total token throughput, mean TPOT), and how concurrency relates to throughput.
• The GLM-5-NVFP4 model: Its architecture with 256 experts, 8 active experts per token, and FP4 quantization.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

1. Sustained decode throughput at 256 output length: 1,876 output tok/s and 2,803 total tok/s at 1,024 requests, with a mean TPOT of 395 ms.
2. Scaling behavior confirmation: Output throughput actually improved with longer outputs (1,876 vs. 1,631 tok/s), confirming that the decode phase benefits from longer sustained runs.
3. Concurrency dynamics: The achieved concurrency of 633 (below the request count of 1,024) indicates that requests were completing faster than they could be injected, suggesting room for higher request rates or different scheduling.
4. A data point for the power puzzle: While the power draw during this specific benchmark was not captured, the improved output throughput suggests that the earlier ~235W reading was at least partially due to short benchmark duration.

Mistakes and Incorrect Assumptions

One potential limitation of this experiment is that the assistant did not simultaneously measure GPU power draw during the longer benchmark. The opening statement "Still ~235W per GPU" refers to the previous measurement, and the new benchmark was run without capturing power data. This means the core question—whether longer benchmarks increase GPU power utilization—remains partially unanswered.

Additionally, the comparison between the 128-output and 256-output benchmarks is not perfectly controlled. The 128-output benchmark was run with --max-running-requests 2048 and --num-continuous-decode-steps 8, while the 256-output benchmark used the same server configuration but with 1,024 requests. The concurrency difference (633 vs. ~1,024) complicates direct comparison.

Conclusion

Message 939 captures a moment of disciplined performance analysis in the midst of a complex optimization campaign. The assistant's observation about GPU power draw, hypothesis about benchmark duration, and subsequent experiment with longer output lengths demonstrate a methodical approach to understanding system behavior. While the results did not definitively resolve the power puzzle, they provided valuable data about sustained decode throughput and confirmed that the system's compute-bound nature persists across different workload characteristics.

This message is a microcosm of the broader optimization effort: each experiment peels back one layer of abstraction, revealing new questions even as it answers old ones. The 28% throughput gain from parameter tuning was real, but the ~235W power draw remained a tantalizing clue that significant headroom might still exist—if only the right combination of kernel optimizations, scheduling strategies, and hardware utilization could be found.