The Power Draw Revelation: Uncovering Blackwell's FP4 Kernel Inefficiency at 235 Watts

Introduction

In the middle of an intensive optimization session for deploying the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message from the AI assistant ([msg 938]) captures a pivotal moment of empirical discovery. After systematically tuning server parameters—raising --max-running-requests to 2048 and setting --num-continuous-decode-steps to 8—the assistant had just achieved a 28% throughput improvement, reaching 2,095 output tokens per second at 2048 concurrency. But the numbers told only part of the story. The assistant needed to understand why the GPUs were performing the way they were, and more importantly, whether the hardware was being fully utilized. The answer would come from a simple nvidia-smi query run during a live benchmark, and the results would reveal a startling gap between what the GPUs could theoretically deliver and what the current software stack was actually achieving.

The Context: A Long Optimization Journey

This message sits at the end of Segment 7 of a much larger session spanning multiple segments of infrastructure setup, driver debugging, virtualization workarounds, and performance tuning. The assistant had been working on deploying the GLM-5-NVFP4 model—a Mixture-of-Experts (MoE) model with 256 experts, 8 active per token—using SGLang with FlashInfer CUTLASS kernels for the FP4 quantized GEMM operations. Earlier in the segment, the assistant had confirmed through TP4+PP2 benchmarking that the model was compute-bound rather than communication-bound, ruling out allreduce latency as the primary bottleneck.

A deep investigation into FP4 GEMM kernel efficiency on SM120 (Blackwell's compute architecture) had revealed that the CUTLASS kernels plateau at approximately 1,300 TFLOPS (70% of dense peak) only for very large matrices. During actual decode operations, per-expert batch sizes of roughly 16–64 tokens achieved merely 0.8–55 TFLOPS—between 0.02% and 3% of peak theoretical compute. The 99KB shared memory limit on SM120 prevented using larger CUTLASS tile configurations (M128×N256 and M256×N128 failed to initialize), fundamentally constraining how much work each kernel invocation could process.

Through systematic tuning—raising --max-running-requests to 2048 and setting --num-continuous-decode-steps 8—the assistant had achieved a 28% throughput improvement at 2048 concurrency, reaching 2,095 output tok/s and 4,151 total tok/s. But this improvement raised a deeper question: were the GPUs finally being pushed to their limits, or was there still headroom left untapped?

The Message: Benchmarking at the KV Cache Wall

The subject message begins with a concise analysis of a benchmark run at 3072 concurrency:

3072 gives 2,062 tok/s — roughly same as 2048. We're KV-cache limited at ~2048 concurrent running requests (495K tokens / 256 tokens per request ≈ 1935 max).

This is a textbook example of identifying a system bottleneck through empirical testing. The assistant had increased concurrency beyond the previous maximum of 2048 to see if throughput would continue scaling. It did not—the throughput plateaued at roughly the same level. The assistant correctly diagnosed this as a KV cache capacity limitation: with 495,488 tokens of KV cache available and each request consuming approximately 256 tokens (128 input + 128 output), the system could sustain at most about 1,935 concurrent requests. Beyond that point, requests would be queued rather than processed in parallel, explaining the throughput plateau.

This analysis demonstrates a mature understanding of the system's resource constraints. The assistant didn't just report the numbers; it connected them to the underlying hardware limitation, showing the reasoning chain from observation to diagnosis.

The Power Draw Investigation

Having established that the KV cache was the limiting factor for concurrency scaling, the assistant pivoted to a different question: were the GPUs actually being utilized efficiently at their maximum sustainable concurrency? To answer this, the assistant designed an experiment that would measure GPU power draw during a live 2048-concurrency benchmark.

The approach was clever: launch the benchmark in the background, wait 15 seconds for it to reach steady-state operation, then query nvidia-smi for power draw, GPU utilization, and SM clock frequency across all eight GPUs. This gave a snapshot of the hardware's actual behavior under realistic inference load.

The results were striking:

0, 241.07 W, 100 %, 2362 MHz
1, 233.94 W, 100 %, 2407 MHz
2, 244.49 W, 100 %, 2317 MHz
3, 234.75 W, 100 %, 2407 MHz
4, 228.82 W, 100 %, 2392 MHz
5, 228.43 W, 100 %, 2377 MHz
6, 236.08 W, 100 %, 2407 MHz
7, 231.35 W, 100 %, 2370 MHz

Every GPU reported 100% utilization and SM clocks in the 2317–2407 MHz range. But the power draw told a different story: each GPU was consuming only 228–244 watts—roughly 38–41% of their 600-watt thermal design power (TDP). This was the critical finding.

What This Reveals About Blackwell's FP4 Performance

The power draw data exposed a fundamental inefficiency in the current software stack. The GPUs were reporting 100% utilization, meaning the SMs (streaming multiprocessors) were never idle—there was always work to do. But they were only drawing 235W out of 600W TDP, which means the work they were doing was not keeping the arithmetic units busy enough to draw full power.

This is a classic symptom of low arithmetic intensity: the kernels are spending most of their time moving data (loading weights, fetching activations, writing results) rather than performing actual floating-point operations. When a GPU's compute units are stalled waiting for data, the SM reports as "active" (it's not idle—it's executing instructions, just not the compute-heavy ones), but the power draw stays low because the FP4 matrix multiply units aren't being fed enough work.

The 100% utilization metric from nvidia-smi is misleading in this context. It measures whether the SMs have work scheduled, not whether the arithmetic pipelines are fully occupied. The true measure of compute efficiency would be achieved occupancy of the FP4 tensor cores, which these power numbers suggest is quite low.

The Thinking Process: Connecting Benchmarks to Hardware Metrics

What makes this message particularly valuable is the thinking process it reveals. The assistant didn't stop at "throughput plateaued at 3072 concurrency." It asked the next question: "Are the GPUs actually working hard?" This is the difference between surface-level benchmarking and deep performance analysis.

The assistant's reasoning chain can be reconstructed as:

  1. Observation: Throughput plateaus at ~2,095 output tok/s beyond 2048 concurrency.
  2. Diagnosis: KV cache capacity limits concurrent requests to ~1,935.
  3. New question: Even at maximum concurrency, are the GPUs fully utilized?
  4. Experiment: Measure power draw during a steady-state benchmark.
  5. Finding: GPUs at 100% utilization but only 235W out of 600W TDP.
  6. Implication: The FP4 kernels are not achieving high arithmetic intensity—the compute units are underfed. This chain shows a systematic approach to performance debugging: identify one bottleneck (KV cache), work around it, then probe deeper to find the next bottleneck (kernel efficiency). The assistant is building a mental model of the system's performance characteristics layer by layer.

Assumptions and Their Validity

The message operates on several assumptions, most of which are sound:

Assumption 1: KV cache capacity is the limiting factor at 3072 concurrency. This is well-supported by the arithmetic: 495,488 tokens / 256 tokens per request ≈ 1,935 requests. The throughput plateau at 3072 confirms this.

Assumption 2: GPU power draw is a meaningful proxy for compute utilization. This is generally valid for NVIDIA GPUs—higher power draw correlates with higher arithmetic intensity and tensor core utilization. However, it's worth noting that power draw can also be affected by clock gating, memory bandwidth utilization, and other factors. The 235W reading is a strong signal but not definitive proof of low arithmetic intensity.

Assumption 3: The 15-second delay before measuring power captures steady-state behavior. This is reasonable for a long-running benchmark with 2048 requests, though the first few seconds might include prefill phases that have different power characteristics than the decode-heavy steady state.

Assumption 4: All 8 GPUs should show similar behavior. The results confirm this—all GPUs are within a narrow 228–244W range, suggesting balanced load distribution across the tensor-parallel configuration.

One potential subtlety: the assistant didn't measure power during the 3072-concurrency run, only during the 2048 run. If KV cache limits concurrency to ~1935, then both 2048 and 3072 runs should have similar numbers of active requests, so the power data should generalize. But it would have been cleaner to measure during the 3072 run as well.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of KV cache in LLM serving: The KV cache stores key-value pairs from previous attention computations, and its size limits how many concurrent requests can be processed. The assistant correctly calculates the maximum concurrency from the cache size and per-request token count.
  2. Knowledge of GPU power metrics: The 600W TDP of the RTX PRO 6000 Blackwell and the significance of operating at 235W (39% of TDP) as an indicator of low compute utilization.
  3. Familiarity with MoE inference characteristics: The GLM-5-NVFP4 model has 256 experts with 8 active per token, meaning each token activates only a fraction of the total parameters. This inherently limits per-expert batch sizes and arithmetic intensity.
  4. Understanding of FP4 quantization and CUTLASS kernels: FP4 (4-bit floating point) GEMM operations have different performance characteristics than FP16 or FP8, and the CUTLASS kernel implementation on SM120 has specific constraints (99KB shared memory limit, tile size restrictions).
  5. Context from earlier benchmarks: The 28% improvement from tuning parameters and the baseline numbers from previous runs.

Output Knowledge Created

This message generates several important pieces of knowledge:

  1. KV cache wall confirmed: The system hits a hard throughput limit at ~2048 concurrent requests due to KV cache capacity. This is a fundamental constraint that can only be addressed by reducing per-request token count, increasing KV cache allocation (if memory permits), or using KV cache compression techniques.
  2. GPU power under inference load quantified: The RTX PRO 6000 Blackwell GPUs draw only 228–244W during FP4 MoE inference, far below their 600W TDP. This provides a concrete target for optimization: if kernel efficiency can be improved, there is 2.5× more power headroom available.
  3. 100% utilization ≠ full compute utilization: The message implicitly demonstrates that nvidia-smi's GPU utilization metric is not a reliable indicator of whether the tensor cores are being fully utilized. This is a crucial lesson for performance engineers.
  4. Validation of the compute-bound hypothesis: The low power draw confirms the earlier analysis that the model is compute-bound (not memory-bandwidth-bound or communication-bound). If the bottleneck were memory bandwidth, power draw would still be relatively low because memory accesses consume less power than compute. If it were communication-bound, GPUs would show idle periods and lower utilization percentages.

The Broader Significance

This message represents a turning point in the optimization session. The assistant has now confirmed that:

Conclusion

The assistant's message at index 938 is a masterclass in empirical performance analysis. It doesn't just report numbers—it connects observations to hardware behavior, tests hypotheses through targeted experiments, and draws actionable conclusions. The discovery that eight RTX PRO 6000 Blackwell GPUs, each with 600W TDP, are collectively drawing under 2,000W while reporting 100% utilization is the kind of finding that redirects an entire optimization effort. It tells the engineer: the hardware is not the problem; the software is not feeding the hardware properly. And that insight is worth more than a thousand benchmark numbers.