The Moment of Synthesis: Diagnosing FP4 Inference Efficiency on Blackwell GPUs

In the middle of an intensive optimization session for the GLM-5-NVFP4 large language model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant produced a message that serves as a diagnostic pivot point. Message [msg 904] is not a tool call, a command execution, or a benchmark result — it is a moment of synthesis. After a long chain of investigations spanning kernel source code analysis, micro-benchmarking, server parameter tuning, and architectural exploration, the assistant pauses to assemble what it has learned into a coherent picture and asks for direction. This message crystallizes the fundamental performance bottleneck that defines the entire optimization effort.

The Context That Led Here

To understand why this message exists, we must trace the path that preceded it. The session had been an odyssey of environment setup, driver installation, kernel compilation, and ultimately model deployment. By the time we reach [msg 904], the assistant has already:

  1. Confirmed the model is compute-bound by benchmarking TP4+PP2 (tensor-parallelism 4 + pipeline-parallelism 2) against TP8 (tensor-parallelism 8). The TP4+PP2 configuration was 2× slower, ruling out allreduce communication latency as the primary bottleneck. If communication were the issue, splitting into smaller TP groups would have improved throughput; instead it degraded.
  2. Investigated the CUTLASS FP4 GEMM kernels in excruciating detail. The assistant examined the generated kernel files in /root/.cache/flashinfer/0.6.3/120a/generated/cutlass_instantiations/120/gemm_grouped/120/, discovering that only two tile configurations work on SM120 (the Blackwell compute architecture): M128×N128×K128 and M128×N128×K256. Two larger tile configurations — M128×N256×K128 and M256×N128×K128 — fail to initialize with the error "Failed to initialize cutlass TMA WS grouped gemm."
  3. Traced the failure to shared memory limits. The assistant dug into the CUTLASS launcher template (moe_gemm_tma_ws_launcher.inl), finding that SM120 uses KernelScheduleAuto and StageCountAutoCarveout, meaning CUTLASS auto-computes how many pipeline stages fit in the available shared memory. With only ~99KB of shared memory per block on SM120 (compared to 228KB on SM100/Hopper), the larger tiles likely require more shared memory than available for even a single pipeline stage.
  4. Measured GPU power draw during inference at 1024 concurrency, finding only ~235W per GPU — a mere 39% of the 600W TDP. Despite 100% GPU utilization reported by nvidia-smi, the GPUs were thermally and electrically underutilized, a classic sign that the kernels being executed are not saturating the compute units.
  5. Run a comprehensive benchmark sweep at concurrency levels of 1, 256, 512, 1024, and 2048, establishing the throughput curve: from 10.2 tok/s at single-stream to 1,640 tok/s at 2048 concurrency, plateauing as the KV cache limit was hit.

What the Message Actually Says

The message opens with a striking observation:

Interesting — power is now ~235W (down from 330W in the previous session's measurements). 100% utilization, ~2400MHz. That's only 39% of the 600W TDP. Even less than before.

This single sentence captures the essence of the problem. The GPUs report 100% utilization, yet they draw barely over a third of their thermal design power. The "utilization" metric from nvidia-smi measures what fraction of the time at least one warp is active on a streaming multiprocessor — it says nothing about how many warps are active or whether the arithmetic units are saturated. The GPUs are "busy" but not "productive" in the sense of doing useful floating-point work.

The assistant then presents a summary table that distills the entire investigation into nine rows:

| Metric | Value | |---|---| | Per-GPU FP4 peak (sparse) | 3,700 TFLOPS | | Per-GPU FP4 peak (dense) | ~1,850 TFLOPS | | Best CUTLASS FP4 micro-bench | ~1,300 TFLOPS (70% dense / 35% sparse) | | GPU power during decode | ~235W / 600W (39%) | | Failing CUTLASS tiles | M128×N256, M256×N128 (larger tiles crash) | | Working tiles only | 128×128×128 and 128×128×256 | | Decode per-expert batch | ~32 tokens at 1024 concurrency |

This table is the output knowledge created by the message — a concise, actionable summary of the state of play. But the reasoning that follows is where the real insight lies.

The Reasoning Process

The assistant's thinking proceeds through several layers of inference:

First, it interprets the spec. The 3.7 PFLOPS number from NVIDIA's marketing is identified as likely the sparse figure, leveraging 2:4 structured sparsity (where half the weights are zero and skipped). The dense ceiling is therefore ~1,850 TFLOPS. This is a crucial distinction: if the team were to enable structured sparsity in the model weights, they could potentially double throughput. But the current model checkpoint (GLM-5-NVFP4) does not use sparsity, so the relevant ceiling is the dense number.

Second, it contextualizes the micro-benchmark results. The CUTLASS kernels achieving 1,300 TFLOPS for large matrices represents 70% of the dense peak — a respectable utilization for any GEMM kernel. But this peak is only reachable when the matrices are large enough to keep the tensor cores saturated. During actual inference, the per-expert GEMMs are tiny.

Third, it performs the critical arithmetic intensity calculation. The model has 256 experts with 8 activated per token. At 1024 concurrent requests, each expert sees approximately 1024 × 8 / 256 = 32 tokens. The resulting GEMM dimensions are roughly 32×2048×6144 (M×K×N in FP4). The assistant calculates the arithmetic intensity at ~16 ops/byte — a staggering 72× below the ~1,158 ops/byte needed to be compute-bound on this hardware. This is the fundamental insight: the model is not communication-bound, it's kernel-bound in a specific way — the GEMMs are too small to utilize the tensor cores efficiently.

Fourth, it identifies the only two mechanisms that can help: grouped/batched GEMM (which FlashInfer already implements) and higher concurrency to increase per-expert batch sizes. Everything else is secondary.

Assumptions and Potential Mistakes

The message makes several assumptions worth examining:

  1. The 3.7 PFLOPS is sparse. This is a reasonable assumption based on NVIDIA's historical marketing practices (they consistently quote sparse TFLOPS for Tensor Core operations). However, the actual dense peak could differ from exactly half depending on the specific FP4 format and hardware implementation. The assistant's estimate of ~1,850 TFLOPS dense is a best guess, not a measured value.
  2. The failing tiles fail due to shared memory limits. The assistant infers this from the error message and the shared memory analysis of the CUTLASS launcher, but hasn't definitively confirmed it by, say, instrumenting the kernel or checking the CUDA error code. The StageCountAutoCarveout logic is complex, and the failure could theoretically stem from other constraints like register pressure or TMA descriptor limits.
  3. cuBLASLt might be faster. The message lists "Different kernel paths — cuBLASLt reportedly works better for small FP4 GEMMs on SM120" as a key lever. This is presented as hearsay ("reportedly") — the assistant hasn't yet tested this claim. In fact, subsequent messages in the session will test cuBLASLt and find it no faster than the CUTLASS path, revealing this assumption to be incorrect.
  4. The per-expert batch calculation is approximate. The formula batch_size * 8 / 256 assumes perfectly uniform distribution of tokens across experts, which is unlikely in practice. Some experts will receive more tokens than others due to the routing policy of the MoE layer. The actual distribution could have a higher variance, meaning some experts see even fewer tokens while a few see more.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message produces several pieces of valuable output knowledge:

  1. A quantified understanding of GPU underutilization: The 235W/600W power draw measurement is a concrete, actionable data point. It tells the optimization team that there is approximately 2.5× headroom in power — if they can find kernels that keep the tensor cores busier, the GPUs can draw more power and deliver more throughput.
  2. A clear bottleneck diagnosis: The problem is not communication (allreduce latency), not memory bandwidth, and not a software bug. It is fundamentally that the per-expert GEMMs during decode are too small to saturate the tensor cores. This rules out many classes of optimization (e.g., NCCL tuning, P2P DMA improvements) and focuses attention on a narrow set of kernel-level interventions.
  3. A ranked set of optimization levers: The message explicitly lists three directions — fixing larger tiles, increasing effective batch size, and trying cuBLASLt. This provides a roadmap for the next phase of work.
  4. A corrected understanding of the hardware spec: The 3.7 PFLOPS figure is contextualized as sparse, giving the team a realistic dense ceiling of ~1,850 TFLOPS against which to measure their 1,300 TFLOPS micro-benchmark result.
  5. A shared vocabulary for the problem: Terms like "per-expert batch size," "arithmetic intensity," and "tile configuration" become the language in which subsequent optimization discussions are conducted.

The Significance of This Moment

Message [msg 904] is the point at which raw data becomes understanding. The assistant has spent many rounds gathering information — running benchmarks, examining kernel source code, measuring power, checking server logs — and now synthesizes it into a coherent narrative. The message's structure mirrors the scientific method: observation (235W power), measurement (throughput at various concurrencies), analysis (arithmetic intensity calculation), and hypothesis (three levers to try).

The message also serves a conversational function: it asks the user for direction. After the synthesis, the assistant offers: "Want me to dig into any of these directions specifically?" This is a handoff point, acknowledging that the next step involves a strategic choice. The user could prioritize fixing the failing tiles (a kernel engineering task), increasing batch size (a server configuration task), or trying cuBLASLt (a backend selection task). Each path leads to different investigations and has different probability of success.

In the broader arc of the session, this message marks the transition from diagnosis to treatment. The first ~70 messages of the segment were about understanding the problem; the messages that follow will be about attempting solutions. The assistant has built a mental model of the system's behavior, identified the root cause, and is now ready to iterate on fixes. The 28% throughput improvement achieved later in the session (reaching 2,095 output tok/s) will come from the "increase effective batch size" lever — specifically, raising --max-running-requests to 2048 and setting --num-continuous-decode-steps 8 — validating the diagnostic framework established in this message.

Conclusion

Message [msg 904] exemplifies the most valuable kind of AI assistant output in a complex engineering context: not the execution of a command, but the synthesis of disparate observations into a coherent understanding. It takes raw data points — power measurements, kernel error messages, benchmark numbers, hardware specs — and weaves them into a causal story about why the GPUs are underperforming. The message's assumptions are clearly stated, its reasoning is transparent, and its conclusions are actionable. For anyone reading this session transcript, this message is the key to understanding why the optimization effort took the direction it did, and what fundamental constraint the team was up against.