The 100% Revelation: How a Single nvidia-smi Command Reshaped the Debugging of GLM-5-NVFP4 on 8 Blackwell GPUs

Introduction

In the course of deploying and optimizing a massive 744-billion-parameter MoE model (GLM-5-NVFP4) across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant issued a seemingly trivial command — a one-liner nvidia-smi query polling GPU utilization, memory usage, clock speeds, and power draw. The output was stark and unambiguous: all 8 GPUs were pegged at 100% utilization, running at maximum clock speeds (2415–2422 MHz) and drawing between 324 and 330 watts each. This single message, [msg 267], was the moment a protracted debugging loop collapsed into a single, crystalline insight. It answered a question that had consumed the preceding dozen messages: What is actually limiting throughput?

This article examines that message in depth: why it was written, what assumptions it overturned, the knowledge it created, and the thinking process that led to it. Though brief in form, the message is dense with significance — a diagnostic fulcrum that pivoted the entire investigation.

Context: The Road to the Question

To understand why this nvidia-smi command was issued, one must trace the reasoning that preceded it. The assistant had successfully deployed GLM-5-NVFP4 using SGLang with --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm, overcoming a persistent NaN crash during decode that had plagued earlier attempts (see [msg 254]). Baseline benchmarks showed approximately 225 output tokens per second and 516 total tokens per second with 64 concurrent requests — respectable numbers, but far below what the raw compute capability of 8 Blackwell GPUs should deliver.

The assistant initially assumed the bottleneck was PCIe bandwidth. This was a reasonable hypothesis: the model uses tensor parallelism across 8 GPUs, which requires all-reduce communication at every layer. With 78 attention layers and 75 MoE layers, each requiring synchronization of hidden states (6144 dimensions in BF16), the total communication volume is substantial. Earlier analysis in [msg 251] and [msg 252] had calculated that all-reduce per token consumed approximately 21 KB, and with batch processing, the total communication per decode step could reach over 100 MB. On a PCIe-based topology without NVLink, this seemed like the obvious bottleneck.

The user then raised a critical observation: PCIe Gen5 x16 should deliver up to 64 GB/s unidirectional bandwidth. If the GPUs were truly PCIe-bound, why was throughput so far below what that bandwidth could support? This prompted an investigation into the actual PCIe topology ([msg 257]–[msg 261]). The assistant discovered that the GPUs were connected via PHB (PCIe Host Bridge) topology — meaning all cross-GPU traffic must go through the CPU and host memory. There was no direct GPU-to-GPU peer-to-peer support (NS status in the topology matrix). The system was running as a KVM/QEMU virtual machine under Proxmox, which added another layer of indirection.

A bandwidth test (referenced in the chunk summary) revealed a startling discrepancy: large transfers achieved approximately 32 GB/s, but small messages typical of all-reduce operations achieved only about 1 GB/s. This 32x gap between achievable and theoretical bandwidth pointed to virtualization overhead as a significant factor. But even this didn't fully explain the throughput ceiling.

The Message Itself: What Was Queried and Why

The command in [msg 267] was:

ssh 10.1.230.175 "nvidia-smi --query-gpu=index,utilization.gpu,utilization.memory,clocks.current.sm,power.draw --format=csv,noheader"

This is a targeted diagnostic query. The assistant deliberately chose five metrics:

  1. utilization.gpu — the fraction of time the GPU was executing kernels. This is the single most important indicator of whether the GPU is compute-bound or waiting on data.
  2. utilization.memory — how busy the memory interface is. High GPU util + low memory util suggests compute-bound; high memory util suggests memory-bandwidth-bound.
  3. clocks.current.sm — the current SM (streaming multiprocessor) clock frequency. If the GPU is throttling due to power or thermal limits, this would show lower-than-maximum values.
  4. power.draw — instantaneous power consumption. Combined with clock speed, this reveals whether the GPU is power-limited. The assistant had attempted to query pci.tx_throughput and pci.rx_throughput in the preceding message ([msg 266]) but discovered those fields were not valid query targets — a small but telling error that forced a pivot to a different set of metrics. The output was:
0, 100 %, 52 %, 2422 MHz, 328.76 W
1, 100 %, 53 %, 2422 MHz, 330.21 W
2, 100 %, 53 %, 2422 MHz, 327.95 W
3, 100 %, 53 %, 2422 MHz, 325.06 W
4, 100 %, 53 %, 2422 MHz, 324.26 W
5, 100 %, 53 %, 2415 MHz, 328.01 W
6, 100 %, 53 %, 2422 MHz, 330.56 W
7, 100 %, 52 %, 2415 MHz, 330.16 W

Every GPU was at 100% compute utilization. Every GPU was running at or near maximum SM clock (2415–2422 MHz). Every GPU was drawing substantial power (324–330W), approaching the TDP of the RTX PRO 6000 Blackwell. Memory utilization was moderate at 52–53%, suggesting the memory subsystem was not the primary bottleneck.

What This Message Revealed

The 100% GPU utilization figure was the key finding. It meant the GPUs were compute-saturated — they were executing kernels continuously, with no idle time waiting for data from other GPUs. This directly contradicted the earlier assumption that PCIe communication was the bottleneck. If the GPUs were stalled waiting for all-reduce results, utilization would show significant idle periods — the GPU would finish its local computation and then wait for data before proceeding to the next layer. Instead, the GPUs were fully occupied.

This finding reshaped the entire performance model. The earlier analysis in [msg 252] had calculated that compute time per token was negligible (approximately 0.18 microseconds per token for MoE layers) compared to communication time (0.86 microseconds). But those calculations assumed idealized FP4 throughput of 418 TFLOPS and did not account for the realities of kernel launch overhead, memory access patterns, and the complex interplay of tensor parallelism with MoE routing. The nvidia-smi data showed that, in practice, the GPU compute units were fully saturated — the model was compute-bound in the real deployment, not communication-bound.

The power readings (324–330W per GPU) further confirmed that the GPUs were under heavy load. The RTX PRO 6000 Blackwell has a TDP likely in the 350–400W range, so drawing 325–330W indicates near-maximum power delivery. The SM clocks at 2415–2422 MHz are consistent with the boost clock under full load — no thermal or power throttling was occurring.

Assumptions Made and Overturned

The most significant assumption that this message overturned was the PCIe-bandwidth bottleneck hypothesis. The assistant had spent considerable effort analyzing all-reduce communication patterns, calculating bytes per token, and comparing TP8 vs EP8 communication volumes — all under the assumption that cross-GPU communication was the primary limiter. The nvidia-smi output showed that the GPUs were not waiting; they were working at full capacity.

A subtler assumption was that compute was negligible relative to communication. The theoretical FLOPs calculations in [msg 252] suggested that MoE compute for a single token required only 0.60 GFLOPs — a trivial amount for a 418 TFLOPS GPU. But this ignored the realities of kernel execution efficiency, memory bandwidth limitations within the GPU, and the overhead of routing and dispatching tokens across experts. The 100% utilization figure revealed that the actual compute efficiency was far lower than theoretical peak, and the GPU was spending its cycles on real work rather than waiting.

The assistant also assumed that PCIe Gen5 x16 bandwidth would be the ceiling. But the bandwidth test had already shown that small-message all-reduce achieved only ~1 GB/s — far below the 64 GB/s theoretical peak. The 100% GPU utilization data added crucial context: even if PCIe were faster, the GPUs were already fully occupied. Faster interconnects would not improve throughput unless they allowed the GPUs to do less work or do it more efficiently.

Input Knowledge Required

To interpret this message correctly, one needs:

Output Knowledge Created

This message created several pieces of actionable knowledge:

  1. The bottleneck is compute, not communication. This was the primary finding. All subsequent optimization efforts should focus on reducing compute load or improving compute efficiency, not on improving cross-GPU communication.
  2. All 8 GPUs are equally loaded. The utilization is 100% across every GPU, with no stragglers. This indicates that the tensor parallelism is well-balanced — no single GPU is disproportionately burdened.
  3. Memory bandwidth is not the primary bottleneck. At 52–53% memory utilization, the memory subsystem has headroom. This suggests that the compute units are the limiting factor, not memory access.
  4. Power and thermal headroom are exhausted. The GPUs are running at maximum clocks and near-maximum power. There is no opportunity for dynamic clock boosting or power capping adjustments to improve throughput.
  5. The throughput ceiling is real. With all GPUs at 100% utilization, the ~225 output tok/s and ~516 total tok/s represent the actual throughput the hardware can deliver for this model under this configuration. Further tuning of server parameters (memory fraction, CUDA graphs, MoE backends) is unlikely to yield significant gains.

The Thinking Process Visible in This Message

The reasoning that led to this message can be reconstructed from the preceding messages. The chain was:

  1. Initial assumption: PCIe bandwidth is the bottleneck (implicit in the EP analysis of [msg 251] and [msg 252]).
  2. User challenge: The user pointed out that PCIe Gen5 x16 should be capable of 64 GB/s, questioning why throughput was so low if PCIe was the limiter ([msg 256] context).
  3. Topology investigation: The assistant checked nvidia-smi topo -m and discovered PHB topology with no P2P support ([msg 257]), then confirmed PCIe Gen5 x16 links ([msg 258]–[msg 261]).
  4. Bandwidth testing: A bandwidth test (referenced in the chunk summary) showed ~32 GB/s for large transfers but only ~1 GB/s for small messages — revealing virtualization overhead as a factor.
  5. The pivot: The assistant attempted to query PCIe throughput directly during inference ([msg 264]–[msg 266]) but hit a dead end when pci.tx_throughput was rejected as an invalid field.
  6. The decisive query: Blocked from measuring PCIe traffic directly, the assistant pivoted to the most fundamental diagnostic: "Are the GPUs actually busy?" The nvidia-smi query in [msg 267] answered that question definitively. This sequence reveals a methodical diagnostic approach: start with a hypothesis (PCIe bottleneck), test it (topology, bandwidth), encounter a contradiction (PCIe is capable but slow for small messages), attempt to measure the hypothesized bottleneck directly (PCIe throughput), fail, and then fall back to the most basic sanity check (GPU utilization). The 100% utilization finding was the breakthrough that reframed the entire problem.

Conclusion

Message [msg 267] is a masterclass in diagnostic minimalism. In a single, eight-line output, it resolved a debate that had consumed a dozen messages and overturned a deeply held assumption about the performance bottleneck. The 100% GPU utilization figure told the assistant and the user something neither had fully appreciated: the RTX PRO 6000 Blackwell GPUs were working at their absolute limit. The model was not communication-bound; it was compute-bound.

This finding had profound implications. It meant that expert parallelism (EP), which the assistant had analyzed extensively in [msg 251] and [msg 252], would not help — EP reduces communication but does not reduce compute. It meant that server-side tuning parameters like memory fraction, CUDA graphs, and MoE backends had already been pushed as far as they could go. And it meant that the ~225 tok/s throughput was not a configuration problem to be solved but a hardware limit to be accepted.

The message also demonstrates the importance of measuring the right thing. The assistant had been deep in theoretical analysis — calculating FLOPs, bytes, and communication ratios — but the real answer came from a simple observation of what the hardware was actually doing. Sometimes the most powerful diagnostic tool is not a sophisticated profiler but a single well-chosen metric that cuts through the noise. In this case, utilization.gpu = 100% was that metric.