The 13% Revelation: Diagnosing Bottlenecks in LLM Inference

In the high-stakes world of large language model deployment, optimization is rarely a straight line. The journey to squeeze every token per second out of a multi-GPU system involves a cascade of tuning parameters, each promising marginal gains, until inevitably the optimizer hits a wall. Message [msg 120] in this opencode session represents precisely that moment — the pivot point where exhaustive NCCL tuning has plateaued, and the assistant must fundamentally re-evaluate where the bottleneck actually lies. This message is a masterclass in diagnostic reasoning: it rejects a naive calculation, constructs a realistic model of memory access patterns, and empirically validates its assumptions, all within a single, tightly-scoped analysis.

The Context: An Optimization Plateau

To understand why this message was written, we must first appreciate what preceded it. The session had been engaged in an extensive NCCL (NVIDIA Collective Communications Library) tuning campaign on an 8-GPU RTX PRO 6000 Blackwell system running the GLM-5 model in GGUF Q4_K quantization. The assistant had systematically tested NCCL_PROTO=LL (achieving ~57.6 tok/s), NCCL_PROTO=LL128 (~45.5 tok/s, worse), default protocol (~47.3 tok/s), SHM CUDA memcpy optimizations (~57.5 tok/s, same as LL), and optimization levels O1 vs O2 (no difference). After this exhaustive search, the performance was firmly stuck at approximately 57.5 tokens per second for single-request decode. Every NCCL knob had been turned, and the needle refused to budge.

This is the classic moment in performance engineering where the practitioner must ask: what haven't I been measuring? The assistant had been optimizing communication, but perhaps communication wasn't the problem. The natural next step was to examine whether the system was memory-bandwidth bound — the most common bottleneck in LLM inference, where the GPU spends most of its time waiting for weights to arrive from HBM (High Bandwidth Memory) rather than computing.

The Reasoning: Two Calculations, One Insight

The message opens with the assistant presenting a deliberately naive calculation, then immediately correcting it. This rhetorical structure — the strawman followed by the refined model — reveals the assistant's thinking process in real time.

The naive calculation is straightforward: the GGUF model file is 402 GB, distributed across 8 GPUs gives 50.2 GB per GPU. At 57.5 tokens per second, if every token required reading all 50.2 GB of weights from HBM, the bandwidth demand would be 50.2 × 57.5 = 2,889 GB/s per GPU. The RTX PRO 6000 Blackwell's HBM3e is rated at approximately 1,500 GB/s. This yields an impossible 192.6% utilization — clearly a nonsensical result. The assistant immediately flags this: "the first calculation (reading all 402GB) is clearly impossible at 57.5 tok/s."

This strawman serves an important purpose. It forces the reader (and the assistant itself) to articulate why the naive model is wrong, which in turn surfaces the architectural properties that make LLM inference efficient. The assistant enumerates two key reasons:

  1. Mixture-of-Experts (MoE) sparsity: The GLM-5 model has 256 experts per MoE layer, but only 8 are activated per token. That means only 8/256 = 3.125% of expert weights are read per forward pass. The remaining ~97% of expert weights sit idle in HBM, consuming power but not bandwidth.
  2. Multi-head Latent Attention (MLA) compression: The model uses MLA, a memory-efficient attention mechanism where the KV cache is compressed through a low-rank projection. This reduces the attention weight footprint compared to standard multi-head attention. The refined calculation builds a detailed model of per-token weight reads. The assistant estimates expert weights at 3 × 6144 × 2048 × 0.5625 bytes per expert (accounting for Q4_K quantization at ~4.5 bits per parameter), then multiplies by 8 active experts plus one shared expert. Attention weights are estimated at roughly 4 × 6144 × 6144 × 0.5625 bytes per layer. Summing across 78 layers and dividing by 8 GPUs yields approximately 3.47 GB read per GPU per token. At 57.5 tok/s, this gives 199.6 GB/s per GPU — just 13.3% of the assumed 1,500 GB/s HBM bandwidth. This is the critical insight: the model is not memory bandwidth bound. At 13% utilization, there is enormous headroom in the memory subsystem. The bottleneck must lie elsewhere — in compute (dequantization kernels, matrix multiplications, attention), in the scheduler/engine overhead, or in some other part of the inference pipeline that hasn't yet been profiled.

The Verification: Empirical Ground Truth

The assistant does not stop at theoretical calculation. Recognizing that the assumed 1,500 GB/s bandwidth figure is just a datasheet specification, the message proceeds to measure it empirically. A simple CUDA benchmark allocates 1 GB of float16 data on GPU 0 and measures the bandwidth of b.copy_(a) — a raw HBM-to-HBM copy. After warmup iterations, 50 timed copies yield a measured bandwidth of 1,458 GB/s, with each copy taking 1.47 ms.

This empirical validation is crucial for two reasons. First, it confirms that the assumed bandwidth is accurate (1,458 vs 1,500 GB/s, within 3%), lending credibility to the utilization calculation. Second, it establishes a baseline for what the hardware is actually capable of, which becomes the reference point for all subsequent bottleneck analysis.

Assumptions and Their Implications

The analysis rests on several assumptions that deserve scrutiny:

Model architecture dimensions: The assistant assumes hidden size 6144, 78 layers, 256 experts with 8 active, intermediate size 2048 for experts and 12288 for the shared expert, and MLA-specific head dimensions. These are derived from the GLM-5 model configuration and are likely accurate, but any error in these dimensions would propagate into the bandwidth calculation.

GGUF Q4_K quantization overhead: The assistant uses 0.5625 bytes per parameter (4.5 bits), which is a reasonable approximation for Q4_K. The actual GGUF Q4_K format uses a block-wise quantization scheme with varying bit allocations, so the true average may differ slightly. However, even a 10-20% error in this figure would not change the qualitative conclusion — the utilization would still be well below 100%.

MoE routing overhead: The calculation assumes exactly 8 experts are read per token. In practice, MoE routing may have some overhead for the gating network itself, and the router's computation is not accounted for. But this is negligible compared to the expert weight reads.

The "all weights read once" model: The refined calculation assumes that each token decode reads each active weight exactly once. In reality, there may be additional reads for activations, KV cache, and intermediate buffers. However, these are typically much smaller than the weight reads and would not materially change the 13% figure.

Input Knowledge Required

To fully appreciate this message, the reader needs:

Output Knowledge Created

This message produces several valuable outputs:

  1. Empirically validated HBM bandwidth: 1,458 GB/s for the RTX PRO 6000 Blackwell, confirming the datasheet specification
  2. Bandwidth utilization estimate: 13.3% for the current workload, definitively ruling out memory bandwidth as the bottleneck
  3. A reusable methodology: The two-stage approach (naive calculation → refined model → empirical validation) is a template for bottleneck analysis in any inference deployment
  4. A redirection of optimization effort: The session can now stop tuning NCCL and memory parameters, and instead focus on compute-bound bottlenecks — dequantization kernels, attention implementations, MoE routing, or scheduler overhead

The Thinking Process: A Window into Diagnostic Reasoning

What makes this message particularly instructive is the visible thinking process. The assistant does not present a polished final analysis; instead, we see the reasoning unfold in real time. The "Wait" that opens the message is a genuine moment of self-correction — the assistant had just computed the naive bandwidth figure in the previous message ([msg 119]) and is now realizing its absurdity.

The structure follows a classic diagnostic pattern:

  1. Observe a symptom: Performance is stuck at 57.5 tok/s despite extensive tuning
  2. Formulate a hypothesis: Maybe we're memory bandwidth bound
  3. Test with a naive model: The 192.6% utilization reveals the model is wrong
  4. Refine the model: Incorporate MoE sparsity and MLA compression
  5. Reach a conclusion: 13.3% utilization → not memory bound
  6. Validate assumptions: Measure actual HBM bandwidth to confirm This is not just a technical analysis; it's a demonstration of intellectual humility. The assistant is willing to present a wrong calculation, recognize its error, and correct it publicly. In the context of an AI coding session, this transparency is invaluable — it builds trust and makes the reasoning process legible to the human user.

Conclusion

Message [msg 120] represents a pivotal moment in the optimization journey. After exhausting the NCCL tuning space, the assistant pivots from communication optimization to bottleneck analysis, using a combination of architectural reasoning and empirical measurement to rule out memory bandwidth as the limiting factor. The 13% utilization figure is a powerful finding: it means the GPUs are spending 87% of their time on something other than waiting for weights. Whether that something is dequantization, attention computation, MoE routing, or scheduler overhead becomes the next frontier for investigation. This message demonstrates that effective optimization is not just about turning knobs, but about knowing which knobs to stop turning — and having the analytical tools to make that determination with confidence.