The Multi-Pronged Investigation: Diagnosing FP4 GEMM Kernel Efficiency on Blackwell SM120

In the high-stakes world of large language model inference optimization, a single user observation can redirect an entire investigation. This is precisely what happens at message 861 of an opencode coding session—a moment where the assistant, having just confirmed a counterintuitive performance result, pivots from configuration tuning to deep kernel analysis. The message is a masterclass in systematic diagnostic methodology: three parallel research subagents dispatched simultaneously to probe a performance mystery from complementary angles.

The Spark: A User's Insight

The immediate context is critical. In the preceding messages, the assistant had been benchmarking the GLM-5-NVFP4 model—a 405-billion-parameter Mixture-of-Experts model quantized to NVFP4 (NVIDIA's 4-bit floating point format)—across different parallelism configurations on a machine with 8× RTX PRO 6000 Blackwell GPUs. The assistant had just completed a TP4+PP2 (Tensor Parallelism 4, Pipeline Parallelism 2) benchmark and found it was roughly 2× slower than the earlier TP8 configuration. At 512 concurrency, TP8 achieved ~2,800 total tok/s while TP4+PP2 managed only ~1,318 total tok/s.

The user's response at [msg 858] cuts directly to the heart of the matter: "Doesn't the halving of perf sort of imply we're compute bound -> kernels not tuned to the GPU?"

This is a remarkably sharp observation. The reasoning is elegant: if the bottleneck were communication (allreduce latency), reducing the number of GPUs participating in each allreduce from 8 to 4—and keeping them NUMA-local—should have improved throughput. Instead, throughput halved. The ~2× slowdown correlates directly with the 2× larger matrix multiplications per GPU (each GPU now processes N/4 instead of N/8 of each layer's GEMMs). This points squarely at compute-bound execution, and more specifically, at the efficiency of the FP4 GEMM (General Matrix Multiply) kernels running on the Blackwell SM120 microarchitecture.

The assistant's response at [msg 859] validates this insight with a detailed analysis, comparing the benchmark results side-by-side and noting three key symptoms: (1) the CUTLASS autotune is picking from available tactics but many SM120-optimal ones may be missing, (2) the 100KB shared memory limit forces smaller tile sizes leading to more memory traffic per FLOP, and (3) the GPUs are only drawing ~330W out of 600W TDP—55% power utilization at 98% SM occupancy, suggesting SMs are stalling on memory rather than doing useful FP4 math.

Message 861: The Multi-Pronged Investigation

Message 861 is the direct follow-through on this analysis. The assistant writes:

Let me do a multi-pronged investigation — check what CUTLASS kernels are available for SM120, what the autotune is selecting, and what alternatives exist:

Then it dispatches three parallel task tool calls, each spawning a subagent that runs its own multi-round conversation to completion. The parent session blocks until all three return. This is a deliberate architectural choice: rather than investigating sequentially (which would take three times as long), the assistant launches all three investigations simultaneously, maximizing the use of wall-clock time.

Task 1: Research SM120 FP4 GEMM Kernels

The first subagent is tasked with researching FP4 GEMM kernel support on SM120 (compute capability 12.0). The prompt asks it to understand why FP4 kernels might be running suboptimally, with key context about the microarchitecture. The subagent returns comprehensive findings about CUTLASS's SM120 support, revealing that CUTLASS has SM120-specific code but it is a "reduced subset of SM100"—meaning Blackwell workstation GPUs get a smaller set of pre-tuned kernels compared to their data-center counterparts (B200/B100). This is a critical finding: the kernel ecosystem for SM120 is less mature, which directly explains why the GPUs are underperforming.

Task 2: Check CUTLASS Autotune Results

The second subagent connects to the remote machine to inspect the actual autotune cache and FlashInfer's FP4 GEMM implementation. It searches for JSON autotune cache files (finding none—the autotune results aren't being persisted), examines the FlashInfer source code for FP4 GEMM implementations, and looks at what CUTLASS tactics are available. This ground-truth investigation is essential: rather than theorizing about what kernels might be available, the assistant is checking what's actually being used on the target system.

Task 3: Research SM120 Tensor Core Specs

The third subagent performs a web search for detailed NVIDIA SM120 tensor core specifications, specifically comparing FP4 tensor core throughput between SM120 (RTX PRO 6000) and SM100 (B200/B100), and investigating whether SM120 has the 5th-generation tensor core features found in data-center Blackwell. This provides the hardware capability baseline—understanding what the hardware is capable of is essential for diagnosing why it's not achieving that potential.

The Thinking Process: Systematic Diagnosis

What makes this message remarkable is the thinking process it reveals. The assistant is not taking a single hypothesis and testing it; it's building a multi-dimensional understanding of the problem space. Each task addresses a different layer of the performance stack:

Kernel layer (Task 1): What CUTLASS kernels exist for SM120? Are there known gaps or limitations?
Runtime layer (Task 2): What is the actual autotune selecting on this machine? Are there cache files, configuration issues, or implementation gaps?
Hardware layer (Task 3): What is the theoretical peak performance of SM120 tensor cores? How does it compare to data-center Blackwell? This three-pronged approach covers the full stack from hardware capability through kernel implementation to runtime behavior. It's a textbook example of systematic performance debugging: gather data from every relevant layer simultaneously, then synthesize to identify the bottleneck.

Assumptions and Input Knowledge

The message rests on several key assumptions, all grounded in the preceding analysis:

The model is compute-bound: The TP4+PP2 benchmark established this. If the model were communication-bound, the opposite performance pattern would have appeared.
FP4 GEMM kernels are the bottleneck: Given that the model is compute-bound and the primary computation is FP4 matrix multiplications (both for attention projections and MoE expert MLPs), the kernels executing these operations are the most likely culprit.
CUTLASS autotune may be selecting suboptimal tactics: The assistant had previously observed TMA WS grouped GEMM kernels with M128/M256 configurations crashing during autotune, suggesting the autotuner is working with a constrained set of viable tactics.
SM120 has unique limitations compared to SM100: The 100KB shared memory limit (vs. larger on data-center GPUs) and the reduced CUTLASS kernel set are known architectural differences that could explain the performance gap. The input knowledge required to understand this message is substantial. One must grasp: the difference between tensor parallelism and pipeline parallelism and why TP4+PP2 doubles per-GPU GEMM sizes compared to TP8; the concept of FP4 quantization and why it requires specialized GEMM kernels; the role of CUTLASS as a kernel generation framework and autotune as a tactic selection mechanism; the SM120 vs. SM100 microarchitecture differences; and the relationship between GPU power draw, SM occupancy, and compute efficiency.

The Significance of Parallel Task Dispatch

A subtle but important architectural detail: the assistant dispatches all three tasks in a single message using the task tool, which spawns subagent sessions. These subagents run to completion independently and in parallel. The parent session blocks until all three return. This is visible in the message structure: the task definitions are followed by their results, all within the same message.

This design choice reveals the assistant's prioritization of efficiency. Rather than investigating sequentially (Task 1 → analyze → Task 2 → analyze → Task 3 → analyze), it launches everything at once. The subagents can each run multiple rounds of conversation—searching files, running commands, browsing the web—without blocking each other. The parent only needs to wait for the slowest subagent to finish.

Output Knowledge Created

The message produces three streams of output knowledge, each valuable independently and collectively essential:

From Task 1: A detailed map of CUTLASS FP4 kernel support on SM120, revealing that it's a reduced subset of SM100. This immediately explains why the autotune has fewer options to choose from and why the selected tactics may be suboptimal.

From Task 2: Ground-truth data about the actual autotune state on the target machine—no persisted cache files, specific FlashInfer FP4 GEMM implementation details, and the set of tactics that are actually viable on SM120.

From Task 3: A hardware capability baseline showing the theoretical FP4 tensor core throughput of SM120 vs. SM100, establishing the upper bound that the kernels should be approaching.

Together, these three data streams enable the assistant to answer the core question: Are the FP4 kernels running at their potential, and if not, why? The answer, which emerges in subsequent messages, is that the kernels are severely underutilizing the hardware—achieving only 0.8–55 TFLOPS during actual decode (0.02–3% of peak) due to small per-expert batch sizes (~16–64 tokens) and the 99KB shared memory limit preventing larger tile configurations.

Conclusion

Message 861 represents a pivotal diagnostic moment in a complex ML inference optimization session. Triggered by a user's incisive observation about compute-bound performance, the assistant launches a coordinated multi-pronged investigation that probes the kernel, runtime, and hardware layers simultaneously. The systematic methodology—parallel subagent dispatch, complementary research angles, and the synthesis of findings across layers—transforms a hypothesis ("the kernels aren't tuned to the GPU") into actionable knowledge about specific SM120 limitations, CUTLASS kernel gaps, and autotune behavior. It's a demonstration of how structured diagnostic thinking, combined with the ability to parallelize research across subagents, can rapidly converge on the root cause of a complex performance problem.