The 87-Watt Revelation: When 3,740 tok/s Masks a Deeper Bottleneck

Introduction

In the middle of an intensive optimization session for deploying the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant issued a seemingly routine diagnostic command. Message 716 ([msg 716]) consists of a single bash invocation — nvidia-smi polling GPU power draw and utilization — followed by its output. On its surface, it is the most mundane of operations: checking whether the hardware is busy. But the numbers that came back told a story that would fundamentally redirect the entire investigation. All eight GPUs were drawing approximately 87 watts each and showing 0% utilization. Given that each RTX PRO 6000 has a 600-watt thermal design power (TDP), this meant the hardware was essentially idling — running at barely 14.5% of its rated capacity — even as the server was achieving an impressive 3,740 tokens per second in total throughput. This single data point exposed a profound disconnect between the throughput numbers the assistant had just celebrated and the actual hardware engagement, and it set the stage for a deep dive into why eight Blackwell GPUs were loafing when they should have been roaring.

The Context: A Triumph That Raised Questions

The moments immediately preceding message 716 were ones of genuine accomplishment. In message 715 ([msg 715]), the assistant had reported a benchmark result of 3,740 tok/s at 1024 concurrency, with a peak output token throughput of 3,945 tok/s. This was a dramatic improvement from the roughly 880 tok/s the system had been achieving earlier in the session. The assistant had unlocked this performance through a series of aggressive optimizations: patching the FlashInfer CUTLASS MoE autotuner to support the SM120 architecture found in Blackwell RTX PRO GPUs, raising --max-running-requests from 64 to 1024, and disabling both CUDA graphs and the radix cache to avoid a crash in the NSA attention backend's page_table_1_flattened code path.

Yet even as the assistant updated its todo list and prepared to push further, a nagging question must have surfaced. The throughput numbers were climbing, but were they climbing enough? Eight RTX PRO 6000 GPUs, each with substantial compute capacity, should theoretically be capable of far more than 3,740 tok/s for a Mixture-of-Experts model. The assistant's stated intent in message 716 reveals this line of thinking explicitly: "Let me check GPU power/utilization during a benchmark to see if we're saturating the hardware, and also run a 2048-concurrency test." The assistant wanted to know whether the GPUs were being fully utilized — whether the throughput ceiling was being set by the hardware's compute limits or by something else entirely.

The Message: A Single Command Carrying Heavy Implications

The full content of message 716 is deceptively brief:

[assistant] Let me check GPU power/utilization during a benchmark to see if we're saturating the hardware, and also run a 2048-concurrency test: [bash] ssh root@10.1.230.174 "nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv,noheader" 0, 0 %, 87.92 W 1, 0 %, 89.24 W 2, 0 %, 85.91 W 3, 0 %, 86.60 W 4, 0 %, 89.38 W 5, 0 %, 88.11 W 6, 0 %, 90.07 W 7, 0 %, 85.52 W

The assistant had planned two actions: check GPU power and run a higher-concurrency benchmark. But the power check alone was so revealing that it became the immediate focus. The command itself uses nvidia-smi with a custom query that requests three fields per GPU: index, GPU utilization percentage, and instantaneous power draw in watts. The output format is CSV with no headers, producing a clean eight-line table.

The numbers are striking in their uniformity. Every GPU reports 0% utilization and a power draw between 85.52 and 90.07 watts. For context, the RTX PRO 6000 Blackwell GPU has a TDP of 600 watts. An idle GPU typically draws somewhere between 50 and 100 watts depending on the specific model, driver state, and memory clock configuration. The 87-watt average across all eight GPUs is consistent with an idle state — the GPUs are powered on, memory is initialized, but no compute kernels are running.

What the Assistant Assumed — and What It Got Wrong

The assistant's reasoning in issuing this command reveals several implicit assumptions. First, the assistant assumed that the impressive throughput numbers being observed might correspond to meaningful GPU utilization. After all, 3,740 tok/s is not a trivial number — it represents substantial work being done. The natural assumption would be that the GPUs are busy processing those tokens.

Second, the assistant assumed that the bottleneck might be at the GPU compute level — that pushing to higher concurrency (2048 requests) would further stress the GPUs and reveal their true ceiling. This assumption is embedded in the stated plan to "also run a 2048-concurrency test."

Third, the assistant assumed that the software stack was functioning correctly — that the MoE kernels, attention backends, and communication primitives were all doing useful work on the GPUs rather than spending most of their time waiting.

The nvidia-smi output shattered all three assumptions. The GPUs were not busy. They were barely awake. The 3,740 tok/s was being achieved despite the hardware being almost completely idle. This meant the bottleneck was not in GPU compute at all — it was somewhere in the data path between the CPU, the network, and the GPUs.

The Deeper Problem: PCIe Bottlenecks and Virtualization Overhead

The 87-watt reading was the first concrete evidence of a problem that the session had been circling for some time. Earlier in the conversation, the assistant had diagnosed that the GPUs were running in a Proxmox virtual machine environment, where each GPU was isolated on its own PCIe root complex. This topology prevented direct peer-to-peer (P2P) DMA transfers between GPUs, forcing all inter-GPU communication to go through the CPU's memory subsystem via PCIe. The assistant had even attempted to migrate to an LXC container to bypass the virtualization overhead, but the NVIDIA driver had failed to initialize due to a GSP firmware incompatibility with the Proxmox host kernel.

The power draw data now confirmed that this PCIe bottleneck was the dominant constraint. The GPUs were spending most of their time waiting for data to arrive over the PCIe bus rather than computing. When a GPU is waiting for data, its power draw drops to near-idle levels because the compute units (CUDA cores, tensor cores) are stalled. The 87-watt reading was the signature of a system where the GPUs were chronically underfed.

Output Knowledge Created by This Message

Message 716 produced a critical piece of diagnostic information that would drive the remainder of the investigation. The output knowledge can be summarized as follows:

Quantified the utilization gap: The GPUs were at 0% utilization and ~87W power draw, versus a 600W TDP. This established a precise upper bound on how much headroom existed — approximately 6x to 7x in power terms.
Refuted the throughput-implies-utilization assumption: The 3,740 tok/s benchmark, while impressive in absolute terms, could not be taken as evidence of efficient hardware usage. The throughput was being achieved despite the GPUs barely working.
Confirmed the PCIe bottleneck hypothesis: The low power draw was consistent with a system where GPUs are stalled waiting for data, providing indirect but strong evidence that the PCIe topology (each GPU on its own root complex, no P2P) was the primary constraint.
Set a new optimization target: The goal was no longer just to increase throughput, but specifically to increase GPU utilization. Any optimization that raised GPU power draw would be a sign that the bottleneck was being addressed.
Exposed the limits of concurrency scaling: Since the GPUs were idle even at 1024 concurrent requests, simply raising concurrency further (to 2048) was unlikely to help. The bottleneck was not request throughput but data movement.

The Thinking Process Visible in the Message

The message reveals a methodical diagnostic approach. The assistant had just achieved a major throughput milestone but immediately asked: Is this actually good? Rather than resting on the accomplishment, the assistant proactively sought a ground-truth measurement of hardware engagement. This is characteristic of a mature optimization methodology — celebrate the wins, but always verify that the hardware is working as hard as it should be.

The phrasing "to see if we're saturating the hardware" is particularly telling. The assistant suspected that the answer might be no, even before running the command. The throughput curve had been climbing with concurrency (689 tok/s at 64, 1,950 at 256, 2,800 at 512, 3,740 at 1024), but the rate of improvement was slowing. A system approaching GPU saturation would show diminishing returns as concurrency increased because the GPUs would eventually become the bottleneck. The fact that throughput was still climbing at 1024 concurrency suggested that the GPUs were not the bottleneck — which is exactly what the power data confirmed.

The assistant also planned a 2048-concurrency test, which would have been the natural next step in a scaling investigation. But the power data made that test somewhat moot — if the GPUs were idle at 1024, they would likely be idle at 2048 too, unless the additional requests somehow changed the computational profile (e.g., by enabling larger batch matmuls that better utilize the tensor cores). The assistant would go on to run the 2048 test anyway in subsequent messages, but the power data had already revealed the fundamental issue.

Conclusion

Message 716 is a masterclass in the importance of measuring the right thing. The assistant could easily have continued optimizing for raw throughput, pushing concurrency higher and higher, chasing ever-larger numbers without ever realizing that the GPUs were barely participating. The nvidia-smi check was a reality check — a moment of diagnostic clarity that reframed the entire problem.

The 87-watt reading transformed the investigation from a throughput optimization exercise into a systems-level bottleneck analysis. The question was no longer "how can we make the model generate tokens faster?" but rather "why are eight of the most powerful consumer GPUs on the market sitting nearly idle while supposedly doing work?" This distinction between apparent performance (throughput) and actual performance (utilization) is one of the most important lessons in systems optimization, and message 716 captures the moment it was learned in this session.