The SM120 Wake-Up Call: When GPU Utilization Lies and Hardware Nuance Matters

Introduction

In the high-stakes world of large language model inference, few moments are as deceptive as the one captured in message 723 of this opencode session. The assistant had just achieved what looked like a stunning victory: pushing the GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts behemoth—to 3,740 total tokens per second across 8 RTX PRO 6000 Blackwell GPUs. GPU utilization was pegged at 100%. Power draw was a healthy ~250W per card. The assistant declared victory, noting that the hardware was "properly saturating."

Then the user spoke.

"Also still low GPU power numbers, ~330W/gpu not 600W even at high batch, debug sglang to figure out what exactly can be at fault, because to it it seems this GPU is definitely not getting fully fed."

This single message shattered the assistant's assumptions, corrected a factual error about TDP, introduced critical hardware knowledge about SM120 vs SM100 architectural differences, and redirected the entire optimization effort. It is a masterclass in domain expertise—a moment where human knowledge about real hardware constraints overrode the AI's overconfident interpretation of surface-level metrics. This article examines message 723 in depth: its reasoning, its assumptions, the knowledge it required and produced, and the profound shift in perspective it forced on the optimization campaign.

The Context: A Victory Built on Shaky Ground

To understand message 723, we must first understand what preceded it. The assistant had been engaged in a multi-session effort to deploy GLM-5-NVFP4—a quantized version of the massive GLM-5 model—on a server with 8 RTX PRO 6000 Blackwell GPUs. The journey had been arduous: resolving NaN crashes during decode, fighting with FlashInfer compilation issues, tuning attention backends, and ultimately moving the server into an LXC container on Proxmox to bypass VFIO/IOMMU PCIe bottlenecks.

By message 718, the assistant had achieved what appeared to be a breakthrough. By enabling FlashInfer CUTLASS MoE autotune for SM120, raising --max-running-requests from 64 to 1024, and disabling cuda graphs and radix cache, the assistant had pushed total throughput from ~880 tok/s to ~3,740 tok/s at 1024 concurrency. Peak output token throughput hit 3,945 tok/s. The assistant checked GPU utilization and found 100% across all 8 GPUs, with power draw at ~250W each. The assistant's summary stated: "~250W per GPU! That's ~2kW total power draw. The RTX PRO 6000 has a 300W TDP, so we're at ~83% TDP. This is much better than the ~85W idle we saw before. The GPUs are actually being used now."

This is the moment before the fall. The assistant was satisfied, even proud. The numbers looked good. The GPUs were "100% utilized." What more could there be?

The Power Paradox: 100% Utilization at 55% Power

The user's opening observation in message 723 is devastating in its simplicity: "~330W/gpu not 600W even at high batch." This single sentence contains two corrections to the assistant's understanding.

First, the user corrects the power reading. The assistant reported ~250W; the user says ~330W. This discrepancy could stem from different measurement points (nvidia-smi reports different values than direct sensor reads), different points in the benchmark cycle, or the user having access to more precise monitoring. Regardless, the user's number is higher—but still far below the target.

Second, and more critically, the user reveals that the TDP is 600W, not 300W. The RTX PRO 6000 Blackwell has a thermal design power of 300W according to some public specs, but the user asserts 600W. This could refer to a different power limit configuration, a factory overclock setting, or simply more accurate knowledge of the specific hardware in use. Whatever the case, the implication is stark: at ~330W, the GPUs are running at only 55% of their thermal budget. The assistant's claim of "83% TDP" was off by a factor of nearly two.

This is the power paradox: nvidia-smi reports 100% utilization, yet the GPUs are drawing barely half their rated power. How can a GPU be fully utilized at half power? The answer lies in what "utilization" actually measures. NVIDIA's GPU utilization metric reflects how busy the compute units are—whether there are active kernels running. But a GPU can be 100% "busy" while doing very little actual computation if it's stalled on memory accesses, synchronization barriers, or PCIe transfers. The utilization metric measures activity, not efficiency. A GPU that's 100% busy waiting for data from another GPU over PCIe will show 100% utilization while drawing far less power than one that's actually churning through matrix multiplications at full clock.

The user's insight is that the GPUs are "not getting fully fed"—they're starved for data. In a tensor-parallel setup across 8 GPUs connected only by PCIe (no NVLink), every single token requires allreduce operations that shuffle activations across the PCIe bus. If the PCIe bandwidth or latency is the bottleneck, the GPUs spend most of their time waiting for data from siblings rather than computing. This manifests as high utilization (because the GPU is "busy" in the kernel) but low power (because the compute units are stalled, not actively multiplying).

SM120 vs SM100: The Hidden Hardware Divide

The user's second major contribution in message 723 is the introduction of critical hardware knowledge: "Note RTX PRO 6000 is slightly different hw than e.g. B200, e.g. smaller shared mem and some other subtle differences we should tune for carefully."

This is a crucial correction to a widespread assumption. The assistant—and much of the open-source inference ecosystem—had been treating "Blackwell" as a monolithic architecture. The code in sglang, FlashInfer, and Triton often checks CUDA_CAPABILITY[0] >= 9 to enable Hopper/Blackwell features, assuming that all compute capability 12.x GPUs have the same characteristics as SM100 datacenter Blackwell (B200). But this is wrong.

The user points to a research document (sm120-attention-fix-research.md) that lays out the differences in stark detail:

| Architecture | Compute Capability | Shared Memory per SM | |---|---|---| | Hopper (H100) | SM90 | 228 KB | | Blackwell Datacenter (B200) | SM100 | 228 KB | | Blackwell RTX (RTX 5090, RTX PRO 6000) | SM120 | 100 KB | | Ampere (A100) | SM80 | 160 KB | | Ampere (RTX 3090) | SM86/89 | 100 KB |

The RTX PRO 6000, despite being a "Blackwell" GPU, has only 100KB of shared memory per SM—less than the A100's 160KB, and less than half of the B200's 228KB. In terms of shared memory capacity, SM120 consumer Blackwell is closer to an RTX 3090 than to a B200.

This has profound implications for the inference stack. Triton kernels that work perfectly on B200 may crash with out-of-resource errors on SM120 because they request more shared memory than available. The extend_attention.py kernel in sglang, for example, uses block sizes of (128, 64) for any GPU with compute capability >= 9, requiring 106KB of shared memory—6KB more than SM120's 100KB limit. The result is a Triton runtime error: "out of resource: shared memory, Required: 106496, Hardware limit: 101376."

The research document the user provides is remarkably thorough. It identifies the specific function (_get_block_sizes_for_extend_attention()), the exact line where the assumption is encoded, the shared memory formula, and the fix: add an SM120-specific case with smaller block sizes like (64, 64) or (32, 64), and reduce num_warps from 8 to 4 for short query lengths. It also references upstream PRs (sglang PR #16975, FlashInfer issues #1147 and #2166, vLLM PR #31089) that have attempted to address SM120 compatibility.

The document goes further to catalog the full range of SM120 issues:

The NVFP4 Hypothesis: Rethinking PCIe Data Transfer

The final insight in message 723 is perhaps the most forward-looking: "If attention is a part of what's sent over pcie consider if we can make it nvfp4 too."

This connects two threads. The first is the PCIe bottleneck: the assistant had already established that cross-GPU communication over PCIe is a major performance limiter. The second is the quantization format: the model weights are already stored in NVFP4 (4-bit floating point), which dramatically reduces memory footprint and bandwidth requirements for weight loading. But the KV cache—specifically the attention key-value data that must be communicated between GPUs during tensor-parallel inference—is still in higher precision.

The user's insight is that if attention data is part of what's being sent over PCIe (and it is—in attention-based models, the KV cache and intermediate activations are communicated during the allreduce steps), then quantizing that data to NVFP4 as well could reduce PCIe bandwidth requirements proportionally. A 4-bit format uses 1/8 the bandwidth of FP32 and 1/4 the bandwidth of FP16. If the attention data can be kept in NVFP4 throughout the inference pipeline, the PCIe bottleneck could be significantly alleviated.

This is a non-trivial engineering challenge. It requires:

  1. Ensuring that the attention computation can operate directly on NVFP4 data (or that the quantization/dequantization overhead doesn't outweigh the bandwidth savings)
  2. Modifying the allreduce implementation to handle NVFP4 tensors
  3. Ensuring numerical accuracy is maintained with the lower precision
  4. Coordinating with the FlashInfer and TRT-LLM backends that handle attention computation But the direction is clear: if the GPUs are starving for data over PCIe, the solution is either to send less data (quantization) or to send it faster (NVLink, which isn't available). Since NVLink isn't an option on these cards, quantization is the only lever.

Assumptions and Corrections

Message 723 exposes several assumptions that the assistant had been operating under:

Assumption 1: 100% GPU utilization means the GPU is fully utilized. The user's correction reveals that utilization metrics measure activity, not throughput. A GPU stalled on PCIe waits shows high utilization but low power and low actual compute throughput.

Assumption 2: The RTX PRO 6000 has a 300W TDP. The user asserts 600W, which changes the assessment from "83% saturated" to "55% saturated." Whether the correct value is 300W or 600W depends on the specific card configuration, but the user clearly has more accurate knowledge of this particular hardware.

Assumption 3: All Blackwell GPUs have similar characteristics. This is the most significant correction. The assistant's entire optimization strategy was built on the assumption that SM120 consumer Blackwell behaves like SM100 datacenter Blackwell. The user's research document shows this is fundamentally wrong—SM120 has less than half the shared memory, different optimal kernel configurations, and different constraints.

Assumption 4: The inference stack is properly configured for the hardware. The assistant had enabled FlashInfer CUTLASS MoE autotune, which was a step in the right direction, but hadn't addressed the attention kernel shared memory issues, the MoE kernel layout issues, or the allreduce fusion incompatibility.

The user's message doesn't just correct these assumptions—it provides the tools to fix them. The research document includes specific code patches, shared memory formulas, and upstream references that the assistant can use to make targeted improvements.

Input and Output Knowledge

Input Knowledge Required

To fully understand message 723, the reader needs:

  1. Understanding of GPU utilization metrics: Knowledge that nvidia-smi's utilization percentage reflects kernel activity, not computational throughput or power efficiency.
  2. GPU architecture knowledge: Familiarity with CUDA compute capabilities (SM90, SM100, SM120), shared memory constraints, and the difference between datacenter and consumer GPU variants.
  3. Triton kernel concepts: Understanding of block sizes (BLOCK_M, BLOCK_N), warps, shared memory formulas, and how these parameters affect kernel resource usage.
  4. Tensor-parallel inference architecture: Knowledge of how MoE models are split across GPUs, the role of allreduce operations, and how PCIe bandwidth limits cross-GPU communication.
  5. Quantization formats: Understanding of NVFP4, FP8, FP16, and how different precision formats affect bandwidth and computation.
  6. The sglang inference stack: Familiarity with attention backends (flashinfer, triton, trtllm), MoE runners, and the server configuration parameters.
  7. The preceding conversation: Knowledge of the assistant's benchmarks, the achieved throughput numbers, and the earlier debugging of PCIe bottlenecks.

Output Knowledge Created

Message 723 produces several valuable knowledge artifacts:

  1. The SM120 hardware compatibility matrix: The research document provides a definitive reference for which SM120 features differ from SM100, including shared memory limits, optimal block sizes, and kernel configuration requirements.
  2. The power-utilization paradox: A clear articulation that 100% GPU utilization with low power draw indicates a data starvation bottleneck, not efficient computation.
  3. The NVFP4 attention hypothesis: A specific optimization direction—quantizing attention data to NVFP4 for PCIe transfer—that could address the root cause of underutilization.
  4. A prioritized debugging agenda: The message implicitly sets priorities: first fix the attention kernel shared memory issues (which may be causing silent performance degradation), then investigate PCIe data quantization.
  5. Upstream reference map: The document links to specific PRs and issues in sglang, FlashInfer, and vLLM, creating a roadmap for SM120 support that the assistant can follow.

The Thinking Process

The user's reasoning in message 723 reveals a sophisticated mental model of the system. Let me reconstruct the likely thought process:

Step 1: Pattern recognition. The user sees the assistant's benchmark results and immediately recognizes a familiar pattern: high utilization, moderate power, good but not great throughput. This pattern screams "PCIe bottleneck" to anyone who has worked with multi-GPU inference on non-NVLink hardware.

Step 2: Hypothesis formation. The user hypothesizes that the GPUs are data-starved. The 100% utilization is a red herring—it just means the GPUs are busy waiting. The real question is: what are they waiting for?

Step 3: Knowledge application. The user knows something the assistant doesn't: the RTX PRO 6000 is SM120, not SM100. This means the shared memory assumptions in the code are wrong. The attention kernels may be using suboptimal block sizes, causing unnecessary shared memory pressure and potentially triggering fallback paths that are slower.

Step 4: Research synthesis. The user has already done the research. The sm120-attention-fix-research.md document isn't created on the fly—it's a pre-existing artifact from previous investigation. The user has likely encountered SM120 issues before, either with this model or another, and has compiled the findings into a reference document.

Step 5: Direction setting. The user provides two concrete directions:

Conclusion

Message 723 is a turning point in this opencode session. It transforms the optimization problem from "tweak parameters to maximize throughput" to "understand the hardware and rebuild the software stack to match." The assistant had been optimizing within the wrong mental model—treating the RTX PRO 6000 as a smaller B200—when in reality it's a fundamentally different architecture with different constraints.

The message's power comes from its combination of correction, knowledge transfer, and direction. It corrects the assistant's misinterpretation of utilization metrics and TDP. It transfers deep hardware knowledge about SM120 vs SM100 differences, complete with a pre-compiled research document. And it sets a clear direction: fix the attention kernels for SM120 shared memory limits, then investigate NVFP4 quantization for PCIe data transfer.

For the reader, this message is a reminder that in systems engineering, surface-level metrics can deceive. 100% GPU utilization doesn't mean the GPU is working hard—it might mean the GPU is waiting hard. The real signal is in the power draw, the PCIe bandwidth counters, and the kernel-level profiling data. And the most important knowledge often comes not from documentation but from engineers who have already fought the hardware and won.

The SM120 wake-up call is this: never assume that because two GPUs share a generation name, they share the same capabilities. Read the fine print. Check the shared memory. And when the utilization says 100% but the power says 55%, start questioning everything.