Peering Under the Hood: Inspecting CUTLASS Tile Configurations for FP4 Inference on Blackwell

Introduction

In the middle of an intensive performance optimization session for the GLM-5-NVFP4 large language model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant pauses the server restart workflow to issue a single, seemingly mundane bash command. The command lists the contents of a cache directory containing autotuned CUTLASS kernel files and extracts their tile configuration parameters. This message—a solitary command with its output—is a pivotal moment of forensic investigation. It is the assistant reaching directly into the compiler's autotuner output to understand why the FP4 matrix multiplication kernels are performing so far below theoretical peak, and in doing so, it uncovers a fundamental hardware limitation that shapes the entire optimization strategy going forward.

The Broader Investigation Context

To appreciate this message, one must understand the investigation that led to it. The assistant and user have been engaged in a multi-day effort to deploy and optimize the GLM-5-NVFP4 model—a Mixture-of-Experts (MoE) transformer using NVIDIA's 4-bit floating point (NVFP4) quantization. The hardware is formidable: eight RTX PRO 6000 Blackwell GPUs, each rated for 4 PFLOPS of FP4 compute. Yet the observed throughput has been stubbornly below expectations.

In the messages immediately preceding this one ([msg 870] through [msg 879]), the assistant conducted micro-benchmarks comparing two FP4 GEMM implementations—sgl_kernel.cutlass_scaled_fp4_mm and FlashInfer's mm_fp4 with the CUTLASS backend. The results were sobering. At the batch sizes that matter during actual decode (16–64 tokens per expert), the FP4 GEMM operations took 30–42 microseconds regardless of problem dimensions, indicating that kernel launch overhead and memory access dominated over actual computation. The measured throughput at batch=16 was approximately 0.8 TFLOPS—a mere 0.02% of the GPU's 4,000 TFLOPS theoretical peak.

The user then interjected ([msg 878]) with a crucial data point: "Specs claim 4 PFLOPS nvfp4." This prompted the assistant to recalculate efficiency numbers ([msg 879]), revealing that even the best-case micro-benchmark achieved only ~1,300 TFLOPS (32.5% of peak). Something was fundamentally limiting kernel throughput.

The assistant then attempted to restart the server with a TP8 configuration ([msg 882]), but the server failed to launch. While waiting for the restart, the assistant began investigating the CUTLASS kernel implementations directly ([msg 883]), listing the generated kernel files in the FlashInfer autotuner cache. It examined one of those files ([msg 884]) to see a single tile instantiation. But a single file only tells part of the story.

What the Subject Message Actually Does

The subject message ([msg 885]) is a single bash command executed over SSH on the remote server:

ssh root@10.1.230.174 'for f in /root/.cache/flashinfer/0.6.3/120a/generated/cutlass_instantiations/120/gemm_grouped/120/*.cu; do echo "=== $(basename $f) ==="; grep -oP "\d+, \d+, \d+, \d+, \d+, \d+" "$f" | sort -u; done'

This command iterates over every .cu file in the autotuner's output directory for SM120 (Blackwell architecture), prints the filename, and extracts all six-integer tuples using a regex pattern. Each tuple represents a CUTLASS tile configuration in the format M, N, K, cluster_x, cluster_y, cluster_z—the dimensions of the tile that the GPU's tensor cores will process in a single kernel invocation.

The output reveals the complete set of tile configurations that survived the autotuner's profiling phase:

Why Tile Configurations Matter

To understand the significance, one must grasp how CUTLASS (CUDA Templates for Linear Algebra Subroutines and Solvers) works. CUTLASS decomposes matrix multiplication into tiles—small sub-matrices that fit in the GPU's shared memory and register files. The tile dimensions (M×N×K) determine how much data reuse the kernel can achieve. A larger tile means more arithmetic operations per byte loaded from memory, which increases computational intensity and moves the kernel closer to the roofline's compute-bound region.

The Blackwell architecture (SM120) introduces several innovations relevant here: fourth-generation Tensor Cores with FP4 support, Tensor Memory Accelerator (TMA) units for asynchronous data movement, and a shared memory size of 99 KB per SM. The tile configurations must respect these hardware constraints.

The six-integer tuples encode the tile geometry: M is the number of rows of the output matrix processed per tile, N is the number of columns, and K is the reduction dimension. The cluster dimensions (1,1,1 in all cases) indicate no clustering—each tile maps to a single SM. The key observation is that all working configurations use M=128. The only configuration that attempted M=256 (the M256_BS_group0 file) produced no matching tuples, meaning either the file contains no valid instantiations or the grep failed.

The Significance of the Findings

The next message in the conversation ([msg 886]) explicitly analyzes what this output means. The assistant identifies four tile configurations that were attempted by the autotuner:

| Tile (M×N×K) | Status | |---|---| | 128×128×128 | Working | | 128×128×256 | Working | | 128×256×128 | FAILS (tactic 14) | | 256×128×128 | FAILS (tactic 15) |

The two larger tile configurations—128×256×128 and 256×128×128—failed during autotuning. These are precisely the configurations that would improve performance by processing larger blocks, increasing data reuse, and reducing the ratio of memory access to computation. The 128×128×128 tile, while functional, is the smallest practical configuration. It processes only 16,384 elements per tile, which for FP4 data (4 bits per element) means each tile handles just 8 KB of weight data. The small tile size forces the kernel to launch many more tiles to cover the full matrix, increasing overhead and reducing the opportunity for data reuse.

The failure of the 256×128 tile is particularly telling. A 256-row tile would double the number of output elements computed per kernel invocation, improving arithmetic intensity. Its failure suggests a hardware limitation—likely the 99 KB shared memory per SM on Blackwell cannot accommodate the combined storage needed for input matrices, output accumulators, and the software pipeline for a 256×128 tile when using FP4 with block scaling factors.

This finding directly explains the poor kernel efficiency observed earlier. With only 128×128 tiles available, the CUTLASS FP4 kernels are fundamentally memory-bound for the problem sizes encountered during MoE decode. Each expert in the GLM-5-NVFP4 model processes a small batch of tokens (typically 16–64), and the weight matrices are approximately 2048×6144. The 128×128 tile is large enough to achieve reasonable efficiency only when the overall problem dimensions are much larger—hundreds or thousands of rows—so that many tiles can be launched concurrently and the GPU's SMs can be fully occupied. During decode, the per-expert batch is so small that the GPU cannot amortize the tile overhead.

Connection to the Larger Performance Story

This message sits at a critical juncture in the optimization journey. The assistant has been systematically working through a hierarchy of potential bottlenecks:

  1. Communication bottleneck: Ruled out when TP4+PP2 was found to be 2× slower than TP8, proving the model is compute-bound.
  2. Kernel implementation choice: FlashInfer's CUTLASS backend was identified as the fastest available FP4 path, but still far from peak.
  3. Tile configuration limitations: Now identified as the root cause—the autotuner cannot use larger tiles on SM120. The discovery that only 128×128 tiles are viable on Blackwell for FP4 MoE GEMMs is a hardware-imposed constraint. It means that no amount of software tuning within the current CUTLASS framework can bridge the gap to the 4 PFLOPS theoretical peak. The assistant must now explore entirely different optimization strategies: expert parallelism to increase per-GPU batch sizes, persistent kernel designs that avoid repeated launch overhead, or alternative GEMM implementations that use different tiling strategies.

Assumptions and Limitations

The assistant makes several implicit assumptions in this investigation. It assumes that the autotuner's generated files represent the complete set of viable kernel configurations—that if a configuration was not generated, it either failed during profiling or was never attempted. This is a reasonable assumption given the autotuner's design, but it is worth noting that the autotuner may have been configured with a limited search space.

The truncated output for the M256 file is a notable gap. The command produces no tuples for cutlass_kernel_file_gemm_grouped_sm120_M256_BS_group0.cu, but the output simply ends—we don't see whether the file exists but contains no matching patterns, or whether the loop iteration was cut off. The assistant does not comment on this truncation in the subject message, though the subsequent analysis ([msg 886]) treats the M256×N128 configuration as a known failure based on earlier autotuner warnings.

Conclusion

This single bash command,平凡的 in its execution, represents the moment when abstract performance numbers crystallize into concrete hardware constraints. By reading the autotuner's output files directly, the assistant bypasses layers of abstraction—benchmarking frameworks, profiling tools, and documentation—to see exactly what the compiler produced. The answer is unambiguous: on Blackwell SM120, the CUTLASS FP4 grouped GEMM kernels are limited to 128×128 tiles because larger configurations fail to initialize. This hardware limitation explains the 0.02% utilization during decode and sets the stage for the creative optimization work that follows—piecewise CUDA graphs, expert parallelism, persistent kernel designs, and other techniques documented in the glb5improvement-xx.md files. Sometimes the most valuable diagnostic tool is not a profiler but a simple for loop and a grep.