The Arithmetic of Disappointment: Diagnosing FP4 Kernel Underutilization on Blackwell GPUs

In the middle of an intense optimization session for the GLM-5-NVFP4 large language model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant produced a message that crystallized weeks of debugging into a single, devastating diagnosis. Message [msg 891] is a moment of analytical clarity: a "Revised Performance Analysis" that explains, with precise arithmetic, why the model's inference throughput is catastrophically lower than the hardware's theoretical peak. It is the kind of message that every performance engineer dreads writing—and needs to write—because it names the problem before it can be solved.

The Context: A Session of Incremental Discovery

To understand why this message exists, one must trace the thread of investigation that led to it. The preceding messages reveal a pattern of systematic probing. The assistant had been running micro-benchmarks comparing sgl_kernel and FlashInfer's CUTLASS FP4 matrix multiplication implementations (<msg id=873-875>), discovering that FlashInfer's path was faster and that both implementations struggled with the tiny batch sizes characteristic of Mixture-of-Experts decode. The user then interjected with a crucial data point: "Specs claim 4 PFLOPS nvfp4" ([msg 878]), correcting the assistant's earlier estimate of ~2,000 TOPS.

This single user correction triggered a cascade of recalibration. The assistant immediately recalculated efficiency numbers at [msg 879], finding that the CUTLASS kernels achieved only 32.5% of the newly-quoted 4,000 TFLOPS peak, and that actual decode workloads reached a mere 0.02% utilization. But the assistant also recognized that the spec number itself needed verification—NVIDIA's marketing often quotes sparse throughput, and the actual dense number could be half. Web searches at [msg 887] confirmed "3.7 Pflops Peak FP4 AI" from a CDW listing, and the assistant inferred the dense peak at ~1,850 TFLOPS.

Meanwhile, the assistant had been investigating the CUTLASS tile configurations that survived autotuning on the SM120 architecture. At <msg id=883-886>, it discovered that only 128×128 tiles were working—the larger M128×N256 and M256×N128 tiles failed during profiling. This was a critical clue: the GPU's 99KB shared memory limit (established in earlier segment analysis) was constraining the kernel's ability to process larger work units, reducing data reuse and increasing memory-bound behavior.

The server itself was in a state of chaos. Multiple restart attempts had failed (<msg id=880-882>), the run script had been lost ([msg 888]), and the assistant had just finished recreating it and launching the TP8 server for the umpteenth time (<msg id=889-890>). With ~90 seconds to wait for the server to load, the assistant had a window to synthesize everything it had learned.

The Message: A Synthesis Under Time Pressure

Message [msg 891] opens with a pragmatic framing: "While the server loads (~90s), let me summarize the revised picture." This is not an idle summary. It is a deliberate analytical pause, a moment to consolidate understanding before the next round of benchmarking. The assistant knows that once the server is up, the focus will shift to empirical throughput testing. This message is the last chance to get the theory right.

The analysis is structured in three layers. First, the assistant establishes the hardware's theoretical limits:

Per-GPU specs: - FP4 peak: 3,700 TFLOPS sparse / 1,850 TFLOPS dense - Memory bandwidth: 1,597 GB/s - Compute intensity needed for compute-bound: 1,850T / 1.597T = 1,158 ops/byte (dense)

This compute intensity calculation is the linchpin. It says that to saturate the GPU's compute units, every byte loaded from memory must fuel 1,158 arithmetic operations. It is an impossibly high bar for small matrix multiplications.

Second, the assistant layers the micro-benchmark results onto this framework:

Our CUTLASS FP4 micro-benchmark results: - Peak achieved: ~1,300 TFLOPS at large sizes → 70% of dense peak (reasonable) - At batch=16 per expert (typical decode): ~0.8 TFLOPS → 0.04% of peak

The contrast is stark. The kernels can achieve 70% of theoretical peak on large, well-shaped matrices. But during actual decode, they achieve 0.04%. This is not a kernel quality problem—it is a workload shape problem.

Third, the assistant connects this to the model's architecture:

The fundamental problem for this model during decode: - 256 experts, 8 activated → each expert sees ~batch_size * 8 / 256 tokens - At 512 concurrent requests: ~16 tokens per expert - At 1024 concurrent: ~32 tokens per expert - These tiny GEMMs (16×2048×6144) have arithmetic intensity ~16 ops/byte — 72x below what's needed to be compute-bound

The arithmetic intensity of 16 ops/byte versus the required 1,158 ops/byte means the GPU is spending nearly all its time waiting for memory transfers. The computation itself is almost free. This is the definition of a memory-bound workload, but with a cruel twist: the memory bandwidth itself is not the bottleneck either. The kernel launch overhead and the inability to amortize it across larger tiles are the true culprits.

The Reasoning Process: Connecting Every Dot

What makes this message remarkable is the chain of reasoning visible in its structure. The assistant does not simply state conclusions. It walks through the arithmetic step by step, showing exactly how each number is derived and what it implies.

The compute intensity threshold of 1,158 ops/byte comes from dividing the dense FP4 peak (1,850 TFLOPS) by the memory bandwidth (1.597 TB/s). This is a standard Roofline model calculation, but its application here is precise. The assistant is implicitly asking: "If I were compute-bound, how many operations would I need to perform per byte of data loaded?" The answer—1,158—is then compared to the actual arithmetic intensity of the decode GEMMs.

The per-expert batch size calculation is equally careful. With 256 experts and 8 activated per token, the tokens are distributed across experts with a load-balancing factor. At 512 concurrent requests, each expert processes roughly 16 tokens. The GEMM dimensions become M=16 (tokens), N=2048 (intermediate dimension), K=6144 (hidden dimension). The arithmetic intensity of such a GEMM is approximately (2×M×N×K) / (M×K + N×K) bytes = (2×16×2048×6144) / (16×6144 + 2048×6144) ≈ 16 ops/byte. The factor of 72× below the compute-bound threshold is then straightforward: 1,158 / 16 ≈ 72.

The assistant also identifies the only two mechanisms that can improve this situation:

The only two things that help at this scale are: 1. Grouped/batched GEMM — fuse all expert GEMMs into one kernel launch (which FlashInfer MoE already does) 2. Higher concurrency — more tokens in flight = larger per-expert batches

This is a realistic assessment. Grouped GEMMs are already being used. Higher concurrency requires more memory for KV cache and more careful scheduling. Neither is a silver bullet.

The Investigation of Failing Tile Configs

The second half of the message pivots to a concrete investigation. The assistant recalls that the M128×N256 and M256×N128 tiles failed during autotuning, and it suspects these could improve performance by enabling better work distribution across SMs. It issues a bash command to inspect the generated kernel file for the M128 tile group that includes the N256 variant:

ssh root@10.1.230.174 'cat /root/.cache/flashinfer/0.6.3/120a/generated/cutlass_instantiations/120/gemm_grouped/120/cutlass_kernel_file_gemm_grouped_sm120_M128_BS_group2.generated.cu'

The output confirms that the kernel instantiation exists in the generated code but was skipped during autotuning due to a profiling failure. The assistant does not yet know why these tiles fail—the shared memory limit is a hypothesis, but the exact constraint (register pressure, TMA descriptor limits, or something else) remains to be determined.

This investigation is important because larger tiles would increase the compute-to-communication ratio of each kernel invocation. A 256×128 tile processes twice as many output elements per threadblock as a 128×128 tile, potentially doubling the data reuse and improving arithmetic intensity. If these tiles could be made to work, they might shift the decode GEMMs from memory-bound toward compute-bound, even if only slightly.

Assumptions and Uncertainties

The message rests on several assumptions that deserve scrutiny. First, the assistant assumes that NVIDIA's quoted 3.7 PFLOPS is the sparse number and that dense throughput is half (1,850 TFLOPS). This is based on NVIDIA's historical marketing practices for sparse tensor operations. If the 3.7 PFLOPS is actually the dense number, then the efficiency picture changes: the peak achieved 1,300 TFLOPS would be only 35% of dense peak, and the compute intensity threshold would be even higher (2,316 ops/byte), making the problem worse. Either way, the qualitative conclusion holds.

Second, the assistant assumes that the CUTLASS kernels are the best available implementation for FP4 GEMM on SM120. This is a reasonable assumption given that FlashInfer and TensorRT-LLM are the primary inference frameworks for this architecture, but there may be vendor-specific kernels (e.g., cuBLASLt FP4 paths) that perform differently. The assistant had tested cuBLASLt earlier and found it no faster than CUTLASS, but this was a limited comparison.

Third, the arithmetic intensity calculation for the decode GEMMs uses the dense operation count (2×M×N×K FLOPs for a matrix multiply). For FP4, the actual FLOP count depends on how the hardware decomposes the 4-bit operations into lower-precision arithmetic. NVIDIA's documentation suggests that FP4 tensor cores perform 2× the operations of FP8, but the exact counting is architecture-specific. The relative comparison (16 vs 1,158 ops/byte) is robust to these details because both numbers would scale proportionally.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several domains:

GPU Architecture: Understanding of tensor cores, FP4 quantization, memory bandwidth, and the Roofline model. The concept of "compute intensity" (ops/byte) and its relationship to the compute-to-memory ratio is essential.

Mixture-of-Experts Models: Knowledge that GLM-5-NVFP4 has 256 experts with 8 activated per token, and that expert routing distributes tokens unevenly across experts. The per-expert batch size is a function of total concurrency divided by the number of experts, scaled by the activation pattern.

CUTLASS and Kernel Autotuning: Familiarity with tile configurations (M×N×K), threadblock clusters, and the autotuning process that selects the best kernel variant for a given architecture. The distinction between working and failing tiles is a key diagnostic signal.

The Preceding Investigation: The micro-benchmark results showing FlashInfer's CUTLASS path outperforming sgl_kernel, the discovery of failing tile configs, and the web research into NVIDIA's spec numbers are all necessary context.

Output Knowledge Created

This message transforms scattered data points into a coherent diagnosis. Before it, the team knew that throughput was lower than expected, that certain tiles failed, and that the spec was 4 PFLOPS. After it, they understand:

  1. The bottleneck is compute, not communication — but not in the usual sense. The compute units are underutilized because the workload shapes are too small to amortize kernel launch overhead and memory access costs.
  2. The gap is quantitative, not qualitative — the kernels achieve 70% of dense peak on large matrices. The problem is workload geometry, not kernel quality.
  3. Only two levers exist — grouped GEMM fusion and higher concurrency. Neither is easy to improve dramatically.
  4. The failing tiles are a concrete target — if the M128×N256 and M256×N128 tiles can be made to work, they would directly improve the compute-to-communication ratio.
  5. The efficiency numbers are sobering — 0.04% of peak during decode is not a bug to be fixed but a fundamental consequence of the model architecture and hardware constraints.

The Broader Significance

Message [msg 891] represents a turning point in the optimization effort. The earlier phases had been about making things work: installing drivers, resolving build issues, getting the model to run without crashing. This message marks the transition to understanding why performance is what it is. It is the moment when the team stops guessing and starts measuring against theory.

The message also demonstrates a valuable engineering practice: using wait time productively. The assistant had ~90 seconds while the server loaded. Instead of idling, it produced a synthesis that would guide the next phase of work. The "while waiting" framing is not an apology—it is a statement of intent. The analysis is important enough to write down even when there is no immediate action to take.

The subsequent messages in the session build directly on this foundation. The assistant goes on to tune server parameters, achieving a 28% throughput improvement, and then documents optimization approaches in a series of glb5improvement-xx.md files. But the core diagnosis from message [msg 891] remains unchanged: the model is compute-bound in the worst possible way, with tiny GEMMs that cannot saturate the GPU's enormous FP4 throughput. The optimizations that follow are all attempts to work around this fundamental constraint—by increasing concurrency, fusing operations, or finding kernel configurations that squeeze more efficiency from the hardware.

In the end, the message is a testament to the value of quantitative reasoning in performance engineering. It does not offer easy answers. It offers a precise description of the problem, grounded in arithmetic that anyone can verify. And that, often, is the most valuable contribution an engineer can make.