The 86ms Gap: Diagnosing FP4 GEMM Overhead in GLM-5 Inference on Blackwell

In the high-stakes world of large language model inference optimization, few moments are as illuminating as the one captured in message 1355 of this opencode session. After weeks of kernel upgrades, CUDA debugging, PCIe tuning, and throughput benchmarking, the assistant finally isolates the root cause of a massive performance discrepancy — and the numbers are stark. This message represents a turning point: the moment speculation gives way to measurement, and the team can finally see exactly where their 95 milliseconds per decode step are going.

The Context: A Performance Mystery

The session had been chasing a perplexing performance gap. The GLM-5-NVFP4 model, deployed across 8 RTX PRO 6000 Blackwell GPUs using SGLang with TP8 tensor parallelism, was achieving only ~10.36 tok/s in single-stream inference. Theoretical analysis had pegged the maximum at ~309 tok/s — a staggering 3.4% efficiency. Something was consuming 86 out of every 95 milliseconds without producing tokens.

Previous work had ruled out the obvious suspects. PCIe bandwidth was fine at ~50 GB/s. The kernel had been upgraded to 6.14.11 with aggressive power management tuning. NUMA balancing was disabled. CPU governor was set to performance. Yet the single-stream throughput remained abysmally low.

The user, growing impatient with broad benchmarks, redirected the assistant in message 1349: "let's focus on 1/2 stream for now, that's glaringly low. stop bench, and write a test util to simulate allreduce transfers and measure latencies/throughput to see if we're dealing with a latency issue or still a compute issue."

The assistant responded by building decode_latency_breakdown.py, a diagnostic tool that simulated each component of a decode step in isolation. Message 1354 ran this tool and produced the raw data. Message 1355 — the subject of this article — is the assistant's analysis of those results.

The Message: A Moment of Clarity

The assistant opens message 1355 with a declaration: "This is extremely revealing." It then presents a concise three-point summary of the diagnostic results:

AllReduce: ~42us per op, 6.6ms total for 156 ops — This is the NCCL latency floor. Even tiny 12KB messages take 42us because of NCCL kernel launch overhead. 156 AllReduces × 42us = 6.6ms of the 95ms budget. That's 7% — significant but not dominant.

>

BF16 GEMMs at M=1: 8-12us each — Very fast because these are BF16 on Blackwell. But the real model uses FP4 GEMMs through CUTLASS grouped GEMM which has much more overhead.

>

Simulated decode (BF16 GEMMs + AllReduces): 8.9ms — Only 9.4% of the real 95ms. The 86ms gap is where all the overhead lives.

This is the critical insight. The simulated decode — which replaces FP4 GEMMs with BF16 equivalents and adds realistic AllReduce latencies — completes in 8.9 milliseconds. The real model takes 95 milliseconds. The difference is 86 milliseconds of unaccounted overhead.

The Reasoning Process: Building Hypotheses

The assistant's thinking is methodical. Having ruled out communication latency (only 7% of the budget) and BF16 compute (negligible), it constructs four hypotheses for where the 86ms gap originates:

  1. FP4 GEMM overhead — The real model uses FP4 grouped GEMM via CUTLASS, which involves complex dispatch logic, tile scheduling, and potentially suboptimal kernel configurations for the SM120 architecture. This is the prime suspect.
  2. MoE routing — With 256 experts, the Mixture-of-Experts routing logic involves top-k selection, permutation, and scattering of tokens to experts. This is implemented in torch-compiled Python, which may have significant overhead.
  3. Attention — The FlashAttention decode kernel and MLA (Multi-head Latent Attention) compression/decompression steps add latency that wasn't captured in the simple GEMM simulation.
  4. Everything else — Layer norms, embeddings, sampling, and other miscellaneous operations that collectively could add up. The assistant doesn't just speculate — it immediately acts, writing a new diagnostic tool called decode_gap_analysis.py to measure the FP4 GEMM and MoE routing costs directly. This is the hallmark of a good debugging approach: narrow the search space iteratively, measuring at each step.

Assumptions and Potential Pitfalls

The analysis in message 1355 rests on several assumptions worth examining:

The simulated decode is representative. The assistant assumes that replacing FP4 GEMMs with BF16 equivalents and adding NCCL AllReduce latencies produces a valid baseline. This is reasonable for isolating the FP4 overhead, but it ignores interactions between components — for example, the actual model's kernel launch schedule may differ from the simulated one, affecting GPU utilization patterns.

AllReduce latency is additive. The assistant sums 156 AllReduce operations × 42μs = 6.6ms, assuming these operations are purely sequential. In practice, the model may overlap communication with computation, or pipeline AllReduces across streams, reducing the effective cost.

The 42μs floor is stable. The NCCL latency measurement of 42μs for small messages may vary with GPU frequency, PCIe congestion, or NUMA effects. The assistant implicitly trusts this measurement as representative.

The gap is entirely in the four listed categories. There's a risk of confirmation bias — having found that communication and BF16 compute are small, the assistant assumes the remaining 86ms must be in FP4 GEMMs, routing, attention, or "everything else." But the gap could also stem from systemic issues like CPU-side kernel dispatch serialization, Python interpreter overhead in the serving stack, or CUDA graph capture failures.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable outputs:

  1. A quantified performance breakdown: 6.6ms AllReduce + ~2.3ms BF16 GEMMs = 8.9ms accounted, leaving 86ms unexplained. This is the first precise measurement of where time goes in this model.
  2. A prioritized investigation roadmap: The four hypotheses give the team a clear order of attack — measure FP4 GEMM overhead first, then MoE routing, then attention, then everything else.
  3. A new diagnostic tool: decode_gap_analysis.py extends the measurement capability to directly probe FP4 GEMM and MoE routing latency, narrowing the search space further.
  4. Confidence that the bottleneck is compute, not communication: This is a crucial finding. If the gap had been in AllReduce, the solution would involve NCCL tuning, faster interconnects, or communication-computation overlap. But with communication ruled out, the focus shifts to kernel optimization — a fundamentally different engineering challenge.

The Broader Significance

Message 1355 exemplifies a critical skill in systems optimization: the ability to decompose a complex performance problem into measurable components. The 86ms gap could have been caused by anything — slow CPU, bad kernel config, NUMA issues, PCIe bottlenecks, NCCL problems, or inefficient kernels. By building a targeted diagnostic tool and running it, the assistant eliminates entire categories of potential causes in a single measurement.

The approach is textbook scientific method: form hypotheses, design experiments that can falsify them, measure precisely, and iterate. The simulated decode experiment falsifies the hypothesis that "communication or BF16 compute is the bottleneck," leaving the team to focus on FP4 GEMM optimization.

For anyone working on LLM inference optimization, this message demonstrates the importance of building micro-benchmarks that isolate specific components. A full-model benchmark tells you that performance is bad. A micro-benchmark tells you why it's bad — and that knowledge is what makes optimization possible.