The Moment of Validation: GPU Utilization Confirmed at 100%

"The GPUs are actually being used now."

In the course of a long and technically grueling optimization session for deploying the GLM-5-NVFP4 model on 8× RTX PRO 6000 Blackwell GPUs, message [msg 718] arrives as a quiet but significant milestone. It is not a dramatic breakthrough — no new code is written, no configuration is changed, no server is restarted. Instead, it is a moment of measurement and validation. The assistant has been driving the system through a series of increasingly aggressive benchmarks, and it pauses to check whether the hardware is actually being utilized. The answer, confirmed by nvidia-smi, is a resounding yes: 100% GPU utilization across all eight GPUs, each drawing approximately 250 watts.

This message is the payoff for a long chain of optimizations. To understand its significance, we must trace the path that led to it.

The Context: From 87 Watts to 250 Watts

Just two messages earlier, at [msg 716], the assistant checked GPU power draw while the server was idle. The result was sobering: all eight GPUs were hovering around 87 watts — barely above their idle floor. This was despite the fact that the server was running and capable of serving requests. The GPUs were powered on, but they were not working. The inference engine was leaving the hardware largely untouched.

The assistant then launched a benchmark at 1024 concurrency and, after a 20-second delay, polled nvidia-smi again. The result, shown in [msg 717], was dramatically different:

0, 100 %, 256.54 W
1, 100 %, 248.95 W
2, 100 %, 255.94 W
3, 100 %, 247.21 W
4, 100 %, 248.81 W
5, 100 %, 249.72 W
6, 100 %, 255.02 W
7, 100 %, 251.61 W

Every GPU was at 100% utilization, drawing between 247 and 256 watts. Message [msg 718] is the assistant's reaction to this data. It processes the numbers, contextualizes them against the RTX PRO 6000's 300W TDP (arriving at ~83% TDP), compares them to the earlier 87W idle reading, and draws the conclusion: the GPUs are finally being used.

Why This Message Matters: The Significance of Utilization

The statement "The GPUs are actually being used now" is deceptively simple. It represents the resolution of a problem that had plagued the entire deployment effort. For a large Mixture-of-Experts (MoE) model like GLM-5-NVFP4 running on eight GPUs with tensor parallelism, achieving high GPU utilization is not automatic. It requires that every component in the inference pipeline — the attention backend, the MoE kernel runner, the communication primitives, the batching scheduler, and the memory manager — work together to keep the GPUs fed with work.

The assistant had been chasing this goal through multiple rounds of optimization. Earlier in the session, throughput had plateaued at around 880 tok/s with GPU power draw languishing far below TDP. The assistant had identified that the FlashInfer CUTLASS MoE autotuner was not running on SM120 (the compute architecture of the RTX PRO 6000 Blackwell GPUs), meaning the MoE kernel selection was suboptimal. By patching model_runner.py to include flashinfer_cutlass in the autotune list and raising --max-running-requests from 64 to 1024, the assistant had unlocked a dramatic throughput improvement — from ~880 tok/s to ~3,740 tok/s at 1024 concurrency (see [msg 714] and [msg 715]).

But throughput alone does not guarantee utilization. A system could achieve high token throughput through clever batching while still leaving GPU compute units idle part of the time. The power draw measurement is a more direct indicator of whether the hardware is being pushed. The jump from 87W to ~250W per GPU confirms that the optimizations are not just moving tokens through the system faster — they are actually keeping the GPUs busy computing.

The Reasoning: Interpreting the Numbers

The assistant's reasoning in this message is worth unpacking. It performs several implicit calculations:

Aggregate power draw: 250W × 8 GPUs = ~2,000W (2kW) total. This is a non-trivial power consumption, confirming the system is under load.
TDP percentage: The RTX PRO 6000 has a 300W TDP (Thermal Design Power), so 250W represents approximately 83% of TDP. The assistant explicitly notes this, which is important because 83% is a healthy utilization level — not quite maxed out, but close. It suggests the GPUs are being exercised without being artificially throttled.
Comparison to idle: The ~85W idle reading from [msg 716] provides the baseline. The delta of ~165W per GPU represents the additional power drawn due to compute activity. This is a clear signal that the optimization work is having the intended effect. The assistant also implicitly validates its earlier decision to disable CUDA graphs and the radix cache. Those changes, which were made to work around a crash bug in the NSA attention path (the page_table_1_flattened attribute error), had the side effect of simplifying the execution model. The benchmark results show that this simplification did not harm performance — in fact, it may have helped by avoiding the overhead of graph management.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message that are worth examining:

That 100% GPU utilization is unambiguously good. While high utilization is generally desirable for throughput-oriented workloads, it can mask inefficiencies. A GPU at 100% utilization could be spending cycles on suboptimal kernels or waiting on memory stalls. The assistant does not yet know whether the utilization is efficient — only that it is high. Later in the session, the assistant would discover that FlashInfer's allreduce fusion was disabled on SM120, meaning the GPUs were working hard but potentially without the benefit of optimized communication kernels.

That power draw is a reliable proxy for useful work. Power draw measures electrical consumption, not necessarily productive computation. A GPU drawing 250W could be executing useful matrix multiplications, or it could be burning power on synchronization overhead or inefficient memory access patterns. The assistant treats the power reading as validation, but it is really just a necessary condition for good performance, not a sufficient one.

That 83% of TDP is "much better" than idle. This is true in a relative sense — a 3× increase in power draw is significant — but it also raises a question: why not 100% of TDP? The remaining 50W per GPU (17% of TDP) represents headroom that is not being utilized. This could be due to PCIe bottlenecks (a known issue from earlier in the session), suboptimal kernel selection, or the missing allreduce fusion. The assistant's framing of "much better" is correct but incomplete.

Input Knowledge Required

To fully understand this message, a reader needs:

Knowledge of GPU power metrics: Understanding that idle power (~85W) vs. loaded power (~250W) is a meaningful signal, and that TDP (300W) provides a reference point for maximum expected power draw under sustained load.
Familiarity with the RTX PRO 6000 Blackwell architecture: The 300W TDP and the SM120 compute architecture are specific to this GPU. The assistant's ability to contextualize 250W as "83% of TDP" relies on knowing this specification.
Understanding of the optimization history: The message references "~85W idle we saw before" ([msg 716]), which only makes sense in the context of the earlier measurement. The reader must know that the assistant had just checked idle power and found it low.
Knowledge of the inference pipeline: The assistant's satisfaction with "100% GPU utilization" is meaningful only if one understands that achieving high utilization for an MoE model with tensor parallelism across 8 GPUs is non-trivial. Communication overhead, load imbalance across experts, and memory bandwidth constraints can all prevent GPUs from reaching full utilization.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

A confirmed baseline: The assistant now knows that the current configuration (FlashInfer CUTLASS autotune, max-running-requests=1024, no CUDA graphs, no radix cache) can drive the GPUs to 100% utilization. This serves as a baseline for future optimization attempts.
A power consumption profile: The ~250W per GPU reading establishes the power envelope of the deployed system. This is useful for capacity planning, cooling requirements, and cost analysis.
Validation of the optimization direction: The assistant's earlier decisions — patching the autotuner, raising concurrency limits, disabling problematic features — are validated by the utilization data. This reinforces the assistant's approach of modifying upstream code to unlock hardware capabilities.
A signal for further investigation: The gap between 250W and 300W TDP (17% headroom) provides a target for future optimization. If the assistant can close this gap, it could potentially achieve another ~20% throughput improvement.

The Thinking Process

The assistant's thinking in this message is concise but reveals a structured analytical process. It begins with the raw data from nvidia-smi (100% utilization, ~250W), immediately contextualizes it against the TDP (300W, ~83%), and then contrasts it with the earlier idle reading (~85W). The exclamation "That's ~2kW total power draw!" shows the assistant thinking in aggregate terms — not just per-GPU metrics but system-level impact.

The final line — "The GPUs are actually being used now" — is almost conversational, but it carries weight. It reflects the assistant's awareness of the journey: the hours spent debugging NaN crashes, patching autotune code, working around attention backend bugs, and tuning server parameters. All of that work was aimed at this goal: making the GPUs work. The message is a checkpoint, a moment to acknowledge that the foundation is solid before moving on to the next challenge.

Conclusion

Message [msg 718] is a brief but pivotal moment in a long optimization session. It is the point where the assistant confirms that its efforts have succeeded in engaging the hardware. The GPUs are no longer idling; they are computing at full capacity. The message demonstrates the importance of measurement in the optimization cycle — without checking nvidia-smi, the assistant would not know whether its configuration changes were actually moving the needle on hardware utilization. It also highlights the gap between throughput metrics and utilization metrics: a system can report high token throughput while leaving GPU compute capacity on the table. Only by measuring both can an engineer be confident that the hardware is being fully exercised.

The work is not done — the 17% headroom to TDP and the missing allreduce fusion for SM120 remain as open challenges. But for this moment, the assistant has earned the right to state, with evidence, that the GPUs are finally doing what they were bought to do.