The Tile That Didn't Fit: Diagnosing FP4 Kernel Underutilization on Blackwell SM120

Introduction

In the high-stakes world of large language model inference, every microsecond counts. When deploying the GLM-5-NVFP4 model—a 5-parameter-scale Mixture-of-Experts (MoE) model quantized to NVIDIA's FP4 format—across eight RTX PRO 6000 Blackwell GPUs, the assistant in this opencode session faced a puzzling performance gap. Despite the hardware's advertised 4 PFLOPS of FP4 compute, actual inference throughput was far below expectations. Message [msg 886] captures a pivotal diagnostic moment: the assistant had just discovered that the CUTLASS kernel autotuner for SM120 (Blackwell's compute architecture) could only compile a subset of tile configurations, and the most promising larger tiles were silently failing. This finding would reframe the entire optimization effort, shifting the focus from communication bottlenecks to fundamental kernel efficiency limitations.

The Path to Discovery

The context leading to this message is essential. Throughout Segment 7 of the session, the assistant had been systematically investigating why the GLM-5-NVFP4 model wasn't achieving its theoretical throughput. Earlier messages ([msg 870]–[msg 876]) showed the assistant benchmarking FP4 matrix multiplication kernels, comparing sgl_kernel's cutlass_scaled_fp4_mm against FlashInfer's mm_fp4 with the CUTLASS backend. The results were sobering: at the per-expert batch sizes typical during autoregressive decode (around 16 tokens), the FP4 GEMM kernels achieved only 0.8 TFLOPS out of a theoretical 4,000—a utilization rate of 0.02%. Even at larger batch sizes, peak efficiency topped out at roughly 32.5%.

The user had interjected at [msg 878] with a crucial data point: "Specs claim 4 PFLOPS nvfp4." This prompted the assistant to recalibrate its efficiency calculations in [msg 879], concluding that the kernels—not inter-GPU communication—were the primary bottleneck. But a critical question remained: why were the FP4 kernels so inefficient on Blackwell hardware?

The Core Finding: Autotuner Survivorship

Message [msg 886] opens with the assistant presenting a concise table of tile configurations discovered by examining the generated CUTLASS kernel files in FlashInfer's autotuner cache. These files, located at /root/.cache/flashinfer/0.6.3/120a/generated/cutlass_instantiations/120/gemm_grouped/120/, contain the compiled kernel variants for the MoE grouped GEMM operation on SM120.

The assistant had previously listed these files in [msg 883], revealing five generated CUDA source files:

Why Tile Size Matters for FP4 Performance

The assistant's analysis of why this matters is concise but profound. In CUTLASS, a tile defines the dimensions of the threadblock's work unit: M (rows of output), N (columns of output), and K (the reduction dimension). Larger tiles improve arithmetic intensity—the ratio of compute operations to memory accesses—because each element loaded from memory is reused more times.

For FP4 data, each matrix element occupies 4 bits (half a byte). The assistant calculates: "Each tile does 128×128×K FLOPs but needs to load 128×K/2 + 128×K/2 bytes of FP4 data." With the working 128×128 tile, the threadblock loads 128×K bytes total (64×K from each matrix) to perform 128×128×K multiply-accumulate operations. The arithmetic intensity is (128×128×K) / (128×K) = 128 operations per byte. With a 256×128 tile, the intensity would double to 256 operations per byte—significantly better.

But the failing tiles aren't just marginally better—they represent the difference between a memory-bound kernel and a compute-bound one. On Blackwell's SM120 architecture, with its 99KB shared memory limit (identified earlier in the segment), the larger tiles likely require more shared memory than available for TMA (Tensor Memory Accelerator) warp-specialized kernels. The error message "Failed to initialize cutlass TMA WS grouped gemm" confirms this: the Tensor Memory Accelerator's warp-specialized kernel launch is failing, almost certainly due to insufficient shared memory for the larger tile's register and shared memory allocations.

The Shared Memory Ceiling

This finding connects directly to the assistant's earlier discovery about the SM120 shared memory limit. In the segment summary, we learn that "the 99KB shared memory limit on SM120 prevents using larger CUTLASS tile configurations (M128×N256 and M256×N128 fail to initialize)." The 128×128×128 tile, with its modest shared memory footprint, fits within the 99KB budget. But the larger tiles—128×256×128 and 256×128×128—each require significantly more shared memory for their accumulator arrays and TMA descriptors, pushing past the limit.

This is a hardware-imposed constraint specific to the Blackwell SM120 architecture. On earlier architectures like SM90 (Hopper), larger shared memory (228KB) would accommodate these tiles. Blackwell's reduced shared memory—a design trade-off likely made to accommodate the new FP4 tensor core and TMA hardware—creates a real limitation for MoE inference workloads that depend on grouped GEMM kernels.

The Pivot to Specification Verification

The second half of message [msg 886] shows the assistant immediately pivoting to verify the hardware specification. Having discovered a concrete kernel limitation, the assistant now needs to understand the theoretical ceiling: "Now let me check if the 4 PFLOPS spec is dense or with sparsity." This launches an Exa web search for the NVIDIA RTX PRO 6000 Blackwell Server Edition specifications.

This pivot is methodologically important. The assistant is establishing a clear chain of reasoning:

  1. The hardware spec claims ~4 PFLOPS FP4.
  2. The working kernels achieve only ~1,300 TFLOPS (32.5% of that).
  3. The failing larger tiles would improve utilization but are blocked by shared memory limits.
  4. But is the 4 PFLOPS figure itself accurate? And is it dense or sparse? The distinction between dense and sparse is critical for NVIDIA GPUs. NVIDIA often quotes peak performance using structured sparsity (2:4 pattern), which doubles the effective throughput. If the 4 PFLOPS figure is the sparse number, the dense peak would be ~2 PFLOPS, and the 1,300 TFLOPS achieved would represent ~65% utilization—still suboptimal but far more respectable.

Assumptions and Reasoning

The assistant makes several implicit assumptions in this message. First, it assumes that the autotuner's generated files represent the complete set of viable tile configurations—that no other tile sizes were attempted and discarded without generating files. This is reasonable given the autotuner's design, but it's worth noting that the autotuner may have been configured with a specific search space that didn't include, say, M=64 or N=64 tiles that might have better shared memory characteristics.

Second, the assistant assumes that the tile failures are due to shared memory limits rather than other factors like register pressure, instruction count, or warp scheduling constraints. The error message "Failed to initialize cutlass TMA WS grouped gemm" is consistent with shared memory exhaustion, but it could also stem from other resource limits.

Third, the assistant assumes that the 4 PFLOPS figure is relevant for the MoE grouped GEMM workload specifically. In practice, peak FP4 throughput is measured on ideal matrix shapes with optimal tile configurations—not on the irregular, small-batch GEMMs that characterize MoE inference.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. A definitive tile configuration map for SM120 MoE FP4 kernels, showing exactly which tile sizes work and which fail.
  2. A clear performance ceiling explanation: the inability to use larger tiles (M=256 or N=256) directly limits arithmetic intensity and keeps the kernels memory-bound.
  3. A prioritized next step: verifying the hardware specification to determine whether the 32.5% utilization is measured against dense or sparse peak.
  4. A diagnostic methodology: the approach of examining autotuner cache files to understand kernel capabilities is itself a reusable technique for GPU performance analysis.

Broader Implications

The tile configuration limitation has cascading effects on the entire optimization effort. If the largest viable tile is 128×128, then for any MoE expert GEMM where the batch size (M) exceeds 128, the kernel must split the work across multiple threadblocks—increasing scheduling overhead and reducing L2 cache hit rates. For the GLM-5-NVFP4 model, which uses top-2 routing with multiple experts, the effective per-expert batch size during decode is typically 16–64 tokens, which fits within M=128. But the N dimension (intermediate size) of 2048 or 6144 would benefit enormously from N=256 tiles.

More broadly, this finding suggests that Blackwell's SM120 architecture, while introducing FP4 tensor core support, may have regressed in some aspects of kernel flexibility compared to Hopper. The reduced shared memory is a deliberate design choice—likely to save area and power—but it creates real friction for MoE inference workloads that depend on grouped GEMM kernels with large tile configurations.

Conclusion

Message [msg 886] is a classic example of the "look under the hood" debugging that characterizes high-performance computing optimization. By examining the autotuner's generated kernel files—the actual compiled code that runs on the GPU—the assistant discovered a hardware-imposed constraint that no amount of server parameter tuning could fix. The Blackwell SM120 architecture's 99KB shared memory limit prevents the use of larger CUTLASS tile configurations, capping FP4 kernel efficiency at roughly one-third of theoretical peak for the working configurations.

This finding reframes the optimization problem: instead of chasing communication bottlenecks or server configuration tweaks, the team must now consider kernel-level solutions—persistent kernels, structured sparsity, or custom FP4 implementations that work within the shared memory budget. The message represents a turning point in the session, where the assistant shifts from "how to tune the server" to "how to work around hardware limitations." It's a reminder that in the age of massive AI models, the smallest hardware details—a 99KB shared memory limit, a failed tile initialization—can determine whether a multi-GPU cluster delivers 800 tok/s or 4,000 tok/s.