The 3.4% Revelation: When Theory Meets Practice in Blackwell FP4 Inference

At the heart of every optimization journey lies a moment of reckoning — the instant when theory confronts reality and the gap between them becomes undeniable. In the opencode session analyzed here, that moment arrives in a single assistant message (index 1201) that is outwardly unremarkable: a brief reflection on a freshly computed theoretical analysis, followed by a shift toward practical next steps. But beneath its casual tone, this message represents a profound turning point in a weeks-long effort to squeeze maximum performance from eight NVIDIA RTX PRO 6000 Blackwell GPUs running the massive GLM-5-NVFP4 mixture-of-experts model. The number that lands with the force of a hammer blow is 3.4% — the fraction of theoretical peak performance the system is actually achieving.

The Long Road to a Number

To understand why this message matters, one must first appreciate the journey that led to it. The user's original question — "for this model on this machine, gen5 pcie, 2 sockets, what's the maximum possible perf in theory, for single stream?" — had been asked in a prior session but left unanswered. The Python calculation script had failed due to a mundane but frustrating shell escaping issue: parentheses in print statements caused zsh to interpret them as glob patterns, producing the cryptic error zsh:1: no matches found: (TP8):. The session had ended with this task incomplete, hanging in the air like an unresolved chord.

When the conversation resumed (at [msg 1195]), the assistant inherited a sprawling context: a 500+ line findings document, twelve improvement proposals, a modified sglang source tree, patched FlashInfer files, and a comprehensive benchmark suite spanning concurrency levels from 1 to 1024. The immediate priority was clear: finally answer the user's theoretical question. The assistant wrote the calculation as a Python file (avoiding the shell escaping trap) and executed it in [msg 1200]. Message 1201 is the response to that execution — the first processing of those results.

The Shock of 3.4% Efficiency

The assistant opens with "Fascinating results" — a restrained understatement for what the numbers actually reveal. The theoretical analysis computed a maximum single-stream throughput of 309 tok/s, derived from a careful accounting of weight bytes loaded per token, HBM bandwidth, and communication overhead across the 8-GPU tensor-parallel configuration. Against this ceiling, the actual measured performance at concurrency 1 was a mere 10.36 tok/s — a utilization rate of 3.4%.

This is the kind of number that stops a team cold. In GPU inference optimization, achieving even 20-30% of theoretical peak is considered respectable; 50% is exceptional. Three point four percent is not just low — it is a signal that something fundamental is broken. The assistant's choice of the word "shocking" (bolded for emphasis in the original) is entirely warranted.

But the message does not dwell on shock. Instead, it immediately pivots to analysis, revealing the assistant's scientific temperament. It notes that the batch scaling table — computed as part of the same theoretical analysis — shows something critical: AllReduce is the bottleneck at high batch sizes, not compute. At batch=256, the theoretical max balloons to ~10,000 tok/s output, yet the system delivers only 718 tok/s. This asymmetry — 3.4% at batch=1 but ~7% at batch=256 — points to different bottlenecks at different operating points.

Self-Correction in Real Time

One of the most revealing aspects of this message is the assistant's willingness to doubt its own model. Immediately after stating that AllReduce is the bottleneck at high batch sizes, it catches itself: "Let me refine this — the AllReduce estimate looks too high because the per-AllReduce latency model is too pessimistic."

This is a critical moment of intellectual honesty. The theoretical model had likely assumed a fixed per-AllReduce latency that was too conservative, making communication look more expensive than it actually is. By flagging this, the assistant demonstrates awareness that theoretical models are only as good as their assumptions — and that the next step must be empirical measurement, not further theorizing.

The re-examination that follows is precise: at batch=1, the total AllReduce data volume across an entire decode step is only 3.3 MB, spread across 156 individual AllReduce operations (one per MoE layer plus attention layers). At ~10 microseconds per operation, that yields 1.56 milliseconds of pure communication latency. In the context of a ~4 millisecond theoretical compute budget (the time needed to stream weights from HBM at ~1,800 GB/s), this 1.56ms represents a significant fraction — roughly half the time budget. This is a genuine insight: even at single-stream, AllReduce latency is a first-order concern, not a negligible overhead.

The 91ms Mystery

The most glaring discrepancy remains unexplained in this message: the gap between the ~4ms theoretical compute-plus-communication budget and the actual 95.14ms measured time-to-first-token. Where does the remaining ~91ms go? The assistant correctly identifies that this requires kernel-level profiling to answer, and proposes doing so. This sets the stage for the diagnostic work that follows in subsequent messages — building latency measurement tools, dissecting the decode step into its components, and ultimately pinning the blame on FP4 GEMM kernel overhead, MoE routing, and attention.

The assistant's thinking here reveals a mature debugging methodology: when the gap between theory and practice is too large to explain by any single factor, the only path forward is measurement. No amount of theoretical refinement will close a 30x gap; only empirical data can reveal where the cycles are actually going.

Assumptions and Their Limits

Several assumptions underpin the assistant's reasoning in this message, some explicit and some implicit:

The AllReduce latency model is assumed to be "too pessimistic" — but this is a hypothesis, not a measurement. The assistant does not yet know the true per-operation AllReduce cost on this specific hardware topology (PCIe Gen5 between NUMA domains, no NVLink). The subsequent discovery that AllReduce accounts for only a few milliseconds of the total decode time validates the suspicion that the model was pessimistic, but also reveals that communication was never the dominant term.

The theoretical compute model assumes perfect HBM bandwidth utilization — that weight streaming can proceed at the full ~1,800 GB/s per GPU. In practice, small GEMM operations (the bane of MoE inference) achieve only a tiny fraction of this bandwidth. The FP4 GEMM benchmark table from earlier findings shows that a 1×768×6144 GEMM — representative of single-token expert computation — achieves just 0.3 TFLOPS, or 0.02% of the 1,850 TFLOPS peak. The theoretical model's assumption of perfect bandwidth utilization is therefore wildly optimistic for the single-stream case.

The EP8 server status assumption is that it might still be running. The assistant says "Let me now check the EP8 server status and get those missing benchmarks" — but the subsequent messages ([msg 1202] and [msg 1203]) reveal that the server has crashed with a cudaErrorIllegalAddress, and moreover that the so-called "EP8 memsafe" scripts don't actually contain --ep-size 8 — they are simply TP8 with reduced memory fraction. This is a significant discovery: the assistant had been operating under the assumption that EP8 (expert parallelism) had been tested, when in fact the configuration was still tensor-parallel-only. This incorrect assumption had propagated through multiple rounds of analysis and documentation.

The Knowledge Flow

This message sits at a critical juncture in the knowledge architecture of the session. It consumes the output of the theoretical analysis script (the 309 tok/s number, the batch scaling table, the AllReduce cost breakdown) and produces several new pieces of knowledge:

A quantified efficiency gap: 3.4% at single-stream, ~7% at batch=256.
A refined understanding of AllReduce cost: 1.56ms across 156 operations at batch=1.
A hypothesis about model pessimism: the AllReduce latency model needs calibration.
A prioritized action plan: profile the kernel-level breakdown, then check EP8.
An updated todo list: theoretical analysis marked complete, EP8 check elevated to in-progress. The message also implicitly creates knowledge by what it does not say. The assistant does not yet know where the 91ms gap comes from — this unknown becomes the driving question for the next several rounds of work. The diagnostic tools built in subsequent messages (decode latency profilers, component-wise timers) are a direct response to the gap identified here.

The Thinking Process on Display

What makes this message particularly valuable for analysis is the visibility it provides into the assistant's reasoning process. Unlike many AI assistant messages that present polished conclusions, this one shows the gears turning in real time:

Stage 1 — Absorption: The assistant takes in the raw result (309 tok/s theoretical, 10.36 actual) and registers its significance with the bolded "shocking."

Stage 2 — Pattern recognition: It immediately scans the batch scaling table and identifies AllReduce as the bottleneck at high batch sizes. This is not a random observation — it reflects deep knowledge of how MoE inference scales: at low batch, compute dominates; at high batch, communication dominates.

Stage 3 — Self-doubt: "Let me refine this — the AllReduce estimate looks too high." This is the most important cognitive move in the message. The assistant recognizes that its own model may be flawed and explicitly flags the need for correction.

Stage 4 — Re-examination: It recalculates from first principles: 3.3 MB total data, 156 operations, ~10us each, 1.56ms total. This ground-up re-derivation validates the concern — the AllReduce cost is significant but not dominant.

Stage 5 — Action planning: Having refined its understanding, the assistant pivots to what needs to happen next: profile the server, check EP8, get the missing benchmarks. The todo list update formalizes this plan.

This five-stage cognitive sequence — absorb, recognize, doubt, re-examine, act — is a hallmark of effective debugging methodology. The assistant does not accept its own output uncritically; it treats the theoretical model as a hypothesis to be tested, not a conclusion to be defended.

The Broader Context

Message 1201 also serves as a narrative pivot point for the entire session. Prior to this message, the work had been largely exploratory: testing various optimization approaches (MSCCLPP, SBO, OEA, EP8), documenting findings, and building up a comprehensive picture of the system's behavior. After this message, the work becomes diagnostic: measuring kernel latencies, building profiling tools, and systematically accounting for every millisecond of the 95ms decode time.

The 3.4% efficiency number acts as a forcing function. It tells the team that none of the optimizations tried so far — not MSCCLPP's ~2% gain, not OEA's marginal benefit, not the 2x improvement from the sglang update — are addressing the root cause. The problem is not at the system level (scheduler, communication, memory management) but at the kernel level (FP4 GEMM efficiency on SM120). This realization, gestating in message 1201, will drive the subsequent deep dive into CUTLASS tile configurations, shared memory limitations, and the fundamental mismatch between SM120's architecture and the small GEMM shapes required by 256-expert MoE models.

Conclusion

Message 1201 is, on its surface, a brief transitional message — a few lines of reflection on a calculation result, followed by a todo list update. But in the context of the full session, it is the moment when the true nature of the problem becomes clear. The 3.4% efficiency figure is not just a number; it is a diagnosis. It tells the team that they are not fighting system-level inefficiencies that can be tuned away with configuration changes, but rather fundamental hardware-software mismatch that will require deep kernel engineering to address.

The message also reveals the assistant's intellectual character: willing to be surprised, quick to self-correct, methodical in its reasoning, and always oriented toward the next actionable step. It is a model of how to respond when theory delivers bad news — not with denial or despair, but with a refined understanding and a clear plan for the work ahead.