The 600W Revelation: How a Simple Power Query Reshaped an Inference Optimization Campaign

The Message

At message index 727 in this coding session, the assistant executed three shell commands to query GPU power characteristics on a remote machine running 8x RTX PRO 6000 Blackwell GPUs:

ssh root@10.1.230.174 "nvidia-smi -q | grep -A5 'Power Readings' | head -20"
ssh root@10.1.230.174 "nvidia-smi --query-gpu=index,power.max_limit,power.default_limit,enforced.power.limit --format=csv,noheader"
ssh root@10.1.230.174 "nvidia-smi -q -i 0 | grep -A20 'Clocks$' | head -25"

The results were deceptively simple:

    GPU Power Readings
        Average Power Draw                             : 84.29 W
        Instantaneous Power Draw                       : 84.29 W
        Current Power Limit                            : 600.00 W
        Requested Power Limit                          : 600.00 W
        Default Power Limit                            : 600.00 W

0, 600.00 W, 600.00 W, 600.00 W
1, 600.00 W, 600.00 W, 600.00 W
2, 600.00 W, 600.00 W, 600.00 W
3, 600.00 W, 600.00 W, 600.00 W
4, 600.00 W, 600.00 W, 600.00 W
5, 600.00 W, 600.00 W, 600.00 W
6, 600.00 W, 600.00 W, 600.00 W
7, 600.00 W, 600.00 W, 600.00 W

    Clocks
        Graphics                                       : 2347 MHz
        SM                                             : 2347 MHz
        Memory                                         : 12481 MHz

On its surface, this is nothing more than a routine hardware interrogation — three nvidia-smi invocations to verify power limits and clock speeds. But in the context of the broader optimization campaign, this message represents a pivotal moment: the moment when a fundamental assumption about the hardware was shattered, and the entire trajectory of the investigation had to be recalibrated.

Context: The Optimization Campaign So Far

To understand why this simple query matters, we need to step back. The session had been a multi-day effort to deploy the GLM-5-NVFP4 model — a massive 744-billion-parameter Mixture-of-Experts language model — across 8 RTX PRO 6000 Blackwell GPUs connected via PCIe (without NVLink). The assistant had already achieved remarkable throughput improvements: from ~880 tok/s to ~3,740 tok/s total throughput by enabling FlashInfer CUTLASS MoE autotune, increasing --max-running-requests, and disabling CUDA graphs and radix cache.

But there was a persistent, nagging problem. During high-concurrency benchmarks, the GPUs were drawing only ~250W each. The user pointed this out in message 725, explicitly asking the assistant to "debug sglang to figure out what exactly can be at fault, because to it it seems this GPU is definitely not getting fully fed." The user also provided a research document (sm120-attention-fix-research.md) outlining the architectural differences between SM120 (consumer Blackwell, used in RTX PRO 6000) and SM100 (datacenter Blackwell, used in B200).

The assistant's response in message 726 showed a critical misunderstanding. It wrote: "250W is well below the 300W TDP (and 600W is what a dual-slot B200 draws, but RTX PRO 6000 TDP is 300W, so ~250W means ~83% TDP which is actually decent)." This assumption — that the RTX PRO 6000 had a 300W TDP — was the lens through which the assistant interpreted all power data up to this point. The ~250W power draw during benchmarks seemed reasonable, even good, at 83% of the presumed TDP.

Message 727 is the direct response to the user's prodding. The assistant set aside its assumption and went to verify the hardware specs directly.

The Revelation

The nvidia-smi output revealed something startling: the RTX PRO 6000 Blackwell GPUs have a 600W TDP per GPU, not 300W. All eight GPUs reported power.max_limit: 600.00 W and power.default_limit: 600.00 W. This means the ~250W observed during benchmarks represented only ~42% of TDP, not the ~83% the assistant had assumed.

This single data point transformed the problem. The GPUs were not "decently utilized" — they were dramatically underutilized. The bottleneck was not a minor tuning issue but a fundamental mismatch between the software stack and the hardware capabilities. With 8 GPUs each capable of drawing 600W, the total system power budget was 4.8 kW, but the inference workload was only consuming about 2 kW. Something was preventing the GPUs from reaching their full compute potential.

The clock speeds confirmed the GPUs were running at their rated frequencies (2347 MHz graphics, 12481 MHz memory), so the underutilization wasn't due to thermal throttling or clock gating. The GPUs simply weren't being given enough work to do.

Assumptions Made and Mistakes Revealed

This message is a masterclass in the danger of assumptions in systems engineering. Several assumptions were at play:

Assumption 1: RTX PRO 6000 TDP is 300W. This was the most consequential error. The assistant conflated the RTX PRO 6000 with consumer-class GPUs like the RTX 4090 (450W) or earlier RTX PRO cards. In reality, the RTX PRO 6000 Blackwell is a workstation-class GPU with a massive 600W TDP, closer to datacenter GPUs than consumer cards. The name "RTX PRO" suggested a prosumer product, but the power envelope revealed its true datacenter-adjacent nature.

Assumption 2: ~250W at 100% utilization means good utilization. The assistant saw utilization.gpu: 100% in earlier nvidia-smi output and interpreted this as full hardware utilization. But GPU utilization percentage in nvidia-smi measures how much of the time the GPU had at least one kernel running — it does not measure how many SMs (Streaming Multiprocessors) were active, nor does it reflect memory bandwidth utilization or compute unit saturation. A GPU can report 100% utilization while only using a fraction of its SMs if kernels are small and serialized.

Assumption 3: The power data was already correct. In message 718, the assistant had already seen ~250W power draw and accepted it without cross-referencing the TDP. The user's challenge in message 725 was necessary to break this complacency.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Knowledge of NVIDIA GPU power monitoring: Understanding that nvidia-smi -q reports per-GPU power readings, that power.max_limit is the maximum power the GPU can draw (the TDP), and that enforced.power.limit is the current cap.
  2. Knowledge of the RTX PRO 6000 product line: Awareness that NVIDIA's workstation GPUs (RTX PRO/Quadro series) often have different TDPs than their consumer counterparts, and that the Blackwell-generation RTX PRO 6000 is a particularly power-hungry card.
  3. Context of the optimization campaign: Understanding that this is a 744B MoE model being served across 8 GPUs, that PCIe is the interconnect (no NVLink), and that the assistant had been iterating on SGLang configuration parameters.
  4. Understanding of GPU utilization metrics: Knowing that utilization.gpu in nvidia-smi is a time-based metric, not a compute saturation metric, and that 100% utilization can coexist with significant underutilization of the GPU's compute capacity.
  5. Familiarity with SM120 vs SM100 differences: The user's research document highlighted that SM120 (consumer Blackwell) has only 100KB shared memory per SM versus 228KB on SM100 (datacenter Blackwell), which constrains kernel design.

Output Knowledge Created

This message produced several critical pieces of knowledge:

  1. Verified TDP of 600W per GPU: This is the foundational fact. All subsequent optimization decisions would be measured against this ceiling.
  2. Confirmed power limits are not artificially constrained: The enforced.power.limit matches the default, meaning no software or firmware cap is limiting power draw. The underutilization is purely a workload issue.
  3. Confirmed clock speeds are nominal: The GPUs are running at their rated frequencies, ruling out thermal or power throttling as causes.
  4. Established a baseline for power efficiency: With ~250W at ~3,740 tok/s total throughput, the system was achieving roughly 15 tok/s per kW per GPU. If the GPUs could be fully utilized to 600W, the theoretical throughput could be 2.4x higher — potentially approaching 9,000 tok/s.
  5. Exposed the gap between perceived and actual utilization: The discrepancy between 100% utilization metric and 42% power draw became the central mystery to solve.

The Thinking Process

The message itself contains no explicit reasoning — it is purely a data-gathering step. But the thinking process is visible in the sequence of commands chosen and the structure of the query:

The assistant starts with nvidia-smi -q | grep -A5 'Power Readings' to get detailed power info for a single GPU. This is the most direct way to check TDP. The -A5 context shows five lines after the match, which captures the key fields: Average Power Draw, Instantaneous Power Draw, Current Power Limit, Requested Power Limit, and Default Power Limit.

Then, recognizing that the first command only shows one GPU, the assistant runs nvidia-smi --query-gpu=index,power.max_limit,power.default_limit,enforced.power.limit --format=csv,noheader to get power limits for all 8 GPUs in a compact format. This confirms uniformity across the system.

Finally, the assistant checks clock speeds with nvidia-smi -q -i 0 | grep -A20 'Clocks$' to verify that the GPU is running at its rated frequencies. This rules out thermal throttling or driver-imposed clock limits.

The progression shows a methodical approach: first verify the power ceiling, then check if it's being enforced, then check if clocks are nominal. Each command builds on the previous one, narrowing down possible causes of underutilization.

Impact on the Investigation

This message fundamentally changed the direction of the optimization campaign. Before it, the assistant was operating under the assumption that the GPUs were ~83% utilized and the remaining performance gains would come from marginal tuning. After it, the assistant knew that the GPUs were only ~42% utilized and that something fundamental was wrong — either the kernels weren't saturating the hardware, the PCIe interconnect was creating a bottleneck that prevented feeding the GPUs fast enough, or the model architecture itself had inherent serialization that limited parallelism.

The 600W TDP revelation also changed the economic calculus. With 8 GPUs each capable of 600W, the system's power budget was 4.8 kW — a significant operational cost. If the inference stack could be tuned to fully utilize this power, the throughput per watt would be dramatically better. If not, the hardware investment was being wasted.

In the subsequent messages, the assistant would go on to investigate FlashInfer allreduce fusion (which was unsupported on SM120), NCCL tuning, decode step batching, and alternative MoE backends — all driven by the recognition that the GPUs had far more headroom than previously believed.

Conclusion

Message 727 appears, at first glance, to be a trivial data-gathering exercise — three nvidia-smi commands that any system administrator could run. But within the narrative of this optimization campaign, it represents the critical moment of reality-checking. The assistant had been operating for multiple rounds under a mistaken assumption about the hardware's power envelope, and it took the user's explicit challenge to prompt this verification.

The lesson is universal in systems engineering: assumptions about hardware specifications are dangerous, especially when dealing with unfamiliar hardware. The RTX PRO 6000 Blackwell, with its 600W TDP, defies easy categorization — it's not a consumer card, not quite a datacenter card, but something in between with power characteristics that surprised even an experienced engineer. The simple act of measuring — of going to the hardware and asking it directly — revealed a gap between perception and reality that reshaped the entire investigation.

In the end, the most powerful tool in the optimization toolkit is not a kernel patch or a configuration change, but the willingness to question one's assumptions and verify them against the raw data the hardware provides.