The 235W Wall: Diagnosing FP4 GEMM Inefficiency on Blackwell SM120

In the sprawling effort to deploy the GLM-5-NVFP4 mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 903] arrives as a moment of reckoning. After dozens of preceding messages spent resolving CUDA toolkit conflicts, wrestling with flash-attn compilation, debugging NaN crashes in NSA decode backends, migrating VMs to enable PCIe P2P, and ultimately achieving a working server in an LXC container with bare-metal GPU topology, the assistant finally has a stable benchmarking environment. The throughput numbers are climbing — but they are plateauing far below what the hardware specifications suggest should be possible. Message [msg 903] is the message where the assistant stares directly at that plateau, measures the power draw, and confronts the uncomfortable truth: the GPUs are barely sipping power while reporting 100% utilization.

The Message in Full

The message begins with a stark admission of diminishing returns:

We're plateauing — 2048 only gives 1,640 tok/s vs 1,601 at 1024. We hit the KV cache limit (max_running_requests=1024).

The assistant then launches a concurrent benchmark at 1024 concurrency while simultaneously sampling GPU power metrics via nvidia-smi. The results are devastating:

0, 237.54 W, 100 %, 2407 MHz
1, 229.30 W, 100 %, 2415 MHz
2, 241.42 W, 100 %, 2332 MHz
3, 226.74 W, 100 %, 2407 MHz
4, 244.72 W, 100 %, 2415 MHz
5, 229.75 W, 100 %, 2407 MHz
6, 239.51 W, 100 %, 2415 MHz
7, 235.87 W, 100 %, 2407 MHz

Every GPU reports 100% utilization and full clock speed (~2400 MHz), yet the power draw hovers around 235W — a mere 39% of the 600W thermal design power (TDP) these Blackwell RTX PRO 6000 cards are rated for. The inter-token latency (ITL) statistics paint a picture of tail-heavy performance: median 264ms is acceptable for a batch workload, but P99 spikes to 3,096ms and the maximum reaches an eye-watering 11,413ms — a full 11-second stall for the unluckiest request.

Why This Message Was Written: The Reasoning and Context

Message [msg 903] sits at a critical juncture in the optimization journey. The preceding messages had systematically narrowed down the performance bottleneck. In [msg 891], the assistant had computed that the FP4 peak per GPU is 3,700 TFLOPS sparse (1,850 TFLOPS dense), and that the CUTLASS kernels achieve ~1,300 TFLOPS for large matrices — a respectable 70% of dense peak. But during actual decode, each expert in the MoE layer sees only ~16–32 tokens (at 512–1024 concurrency), and the resulting tiny GEMMs achieve a paltry 0.8–55 TFLOPS, or 0.02–3% of peak.

The assistant had also discovered that the larger CUTLASS tile configurations (M128×N256 and M256×N128) fail to initialize on SM120 due to shared memory constraints — the 99KB shared memory limit on Blackwell is insufficient for these tiles when using the auto-selected kernel schedule. This leaves only 128×128 tiles working, which fundamentally limits compute density.

By [msg 903], the assistant has pushed concurrency from 256 through 2048 and watched throughput scale sub-linearly: 704 tok/s at 256 concurrency, 1,106 at 512, 1,601 at 1024, and only 1,640 at 2048 — essentially flat after 1024. The KV cache limit (max-running-requests=1024) is the immediate ceiling, but the deeper question is why the GPUs are so inefficient even within that window.

The power measurement is the key diagnostic. 235W at 100% utilization means the GPU's compute units are active but not doing useful work efficiently. This is the signature of a memory-bound or instruction-starved workload masquerading as compute-bound. The SMs are busy — they're 100% utilized — but they're spending their cycles on low-arithmetic-intensity operations, waiting on data movement rather than performing dense matrix math. The power draw is low because the tensor cores are not being fed enough work per instruction; the GPU is "churning" rather than "burning."

Input Knowledge Required to Understand This Message

To fully grasp the significance of message [msg 903], a reader needs substantial background knowledge spanning several domains:

Hardware architecture: Understanding that GPU power draw is a proxy for compute efficiency — a GPU drawing 235W out of 600W TDP while reporting 100% utilization is in a pathological state where the SMs are active but the tensor cores are underutilized. The 600W TDP represents the thermal design point for sustained FP4 matrix multiplication at peak throughput; drawing only 39% of that means the actual FLOP/s delivered is far below theoretical peak.

Mixture-of-Experts inference: The GLM-5-NVFP4 model has 256 experts with 8 activated per token. Each token's hidden state is dispatched to 8 experts, each performing a GEMM of shape [batch_per_expert, hidden, intermediate]. At 1024 concurrent requests with typical decode batch sizes, each expert sees roughly 32 tokens. A 32×2048×6144 GEMM has extremely low arithmetic intensity — roughly 16 operations per byte loaded — far below the ~1,158 ops/byte needed to saturate the FP4 tensor cores on Blackwell.

CUTLASS tile configurations and shared memory: The failing M128×N256 and M256×N128 tiles would have allowed better work distribution across SMs and higher data reuse, but they require more shared memory than the 99KB available on SM120. The KernelScheduleAuto combined with StageCountAutoCarveout in the CUTLASS template fails to fit even a single pipeline stage for these larger tiles.

Benchmarking methodology: The sglang.bench_serving tool with --dataset-name random --random-input-len 128 --random-output-len 128 generates synthetic requests with fixed input and output lengths, allowing controlled measurement of throughput and latency at varying concurrency levels. The --request-rate 999 flag effectively disables rate limiting, creating a burst of concurrent requests that saturates the server.

The Thinking Process Visible in the Message

The assistant's reasoning in [msg 903] is compressed but revealing. The opening line — "We're plateauing" — acknowledges that the scaling curve has flattened. The immediate diagnosis is the KV cache limit (max_running_requests=1024), but the assistant doesn't stop there. By launching a concurrent benchmark and a power measurement in the same shell command, the assistant is performing a differential diagnosis: is the bottleneck in the scheduler (KV cache slots) or in the compute kernel itself?

The choice to sample power during a 1024-concurrency run (rather than 2048) is deliberate. At 1024 concurrency, the server is within its configured limits and should be operating at peak efficiency. If the GPUs are still drawing only 235W at this point, the problem is not the scheduler ceiling but the kernel efficiency floor.

The power data arrives and confirms the worst hypothesis. The assistant then captures ITL statistics showing the tail latency distribution. The gap between median (264ms) and P99 (3,096ms) — a factor of nearly 12× — indicates that some requests are getting stuck, likely due to scheduling imbalances in the MoE dispatch or straggler experts in the grouped GEMM.

What's not said in this message is equally important. The assistant does not immediately propose a fix. This message is purely diagnostic — it gathers the evidence that will inform the optimization strategies explored in subsequent messages. The silence after the data is the sound of a system being understood.

Output Knowledge Created by This Message

Message [msg 903] creates several critical pieces of knowledge that reshape the optimization effort:

  1. Empirical confirmation of the power ceiling: The GPUs draw ~235W under full decode load, confirming that the FP4 GEMM kernels are not achieving the compute density needed to approach the 600W TDP. This rules out thermal throttling or power capping as explanations and points squarely at kernel efficiency.
  2. Quantified plateau behavior: Throughput scales from 704 tok/s (256 concurrency) to 1,640 tok/s (2048 concurrency) but flattens after 1024. This establishes the diminishing returns curve and sets a baseline for measuring future improvements.
  3. Tail latency characterization: The ITL distribution — median 264ms, P95 314ms, P99 3,096ms, max 11,413ms — reveals that while median performance is acceptable, tail performance degrades catastrophically. This is a critical quality-of-service metric for any production deployment.
  4. The KV cache limit as an immediate constraint: The plateau at 1024 concurrency is partly artificial — the server's --max-running-requests 1024 parameter caps the number of concurrent decode sequences. This is a tunable parameter, and raising it becomes an obvious next step.

Assumptions and Potential Mistakes

The message makes several implicit assumptions worth examining:

Assumption that 100% utilization means the GPU is fully occupied: NVIDIA's reported "utilization" measures the fraction of time that at least one SM is active. It does not measure tensor core utilization or memory bandwidth utilization. A GPU can show 100% utilization while achieving only a fraction of its peak FLOP/s if the SMs are spending most of their cycles on low-throughput instructions or waiting on memory. This is exactly what the power data reveals.

Assumption that the benchmark workload is representative: The random-input-len 128 / random-output-len 128 configuration creates a homogeneous workload. Real production traffic would have variable input lengths, output lengths, and request arrival patterns. The plateau behavior might differ under more realistic conditions.

Assumption that power draw is a reliable proxy for compute efficiency: While generally true, power draw can be affected by many factors including voltage-frequency curve positioning, temperature, and the specific mix of instructions executed. The 235W reading is a strong signal but not a definitive measurement of FLOP/s.

Potential mistake: conflating KV cache limit with kernel efficiency limit: The assistant correctly identifies both constraints but the message structure — presenting the plateau first, then the power data — could be read as implying they're the same problem. They are related but distinct: the KV cache limit caps the number of concurrent sequences, while the kernel efficiency limits the throughput per sequence.

The Broader Significance

Message [msg 903] is the turning point in segment 7 of this coding session. Before this message, the assistant had been exploring whether the bottleneck was communication (allreduce latency), scheduling (KV cache slots), or kernel configuration (CUTLASS tile sizes). After this message, the focus shifts decisively to kernel-level optimization: raising --max-running-requests, tuning --num-continuous-decode-steps, exploring expert parallelism, piecewise CUDA graphs, MSCCLPP allreduce, and persistent grouped GEMM kernels — all documented in the glb5improvement-xx.md files that follow.

The 235W measurement is the fulcrum. It transforms the optimization problem from "how do we feed the GPUs more work?" to "how do we make each unit of work use the GPUs more efficiently?" — a subtle but profound shift in strategy. The assistant will go on to achieve a 28% throughput improvement in subsequent messages, reaching 2,095 output tok/s at 2048 concurrency, but the fundamental challenge of FP4 GEMM efficiency on SM120 remains the defining constraint of this deployment.

In the end, message [msg 903] is a masterclass in diagnostic reasoning: a single concurrent measurement that cuts through the complexity of distributed inference optimization and reveals the true nature of the bottleneck. The GPUs are running at 100% utilization, drawing 235W, and delivering 1,640 tok/s — and understanding why those three numbers coexist is the key to everything that follows.