Discovering SM120-Specific CUTLASS MoE Kernels: A Turning Point in Blackwell GPU Optimization

Introduction

In the course of deploying the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant reached a critical juncture in message [msg 283]. This message represents a discovery moment — the point at which the assistant transitioned from trying generic MoE (Mixture-of-Experts) kernel backends toward exploiting the GPU architecture's specific capabilities. The message is a pure investigation: no decisions are finalized, no configuration is applied, but the knowledge foundation for all subsequent optimization is laid.

Context and Motivation

To understand why this message was written, one must trace the thread of failures and frustrations that preceded it. The assistant had been wrestling with the GLM-5-NVFP4 model — a massive MoE architecture with 256 experts — on eight RTX PRO 6000 Blackwell GPUs (SM120 architecture). After resolving a critical NaN crash during decode by selecting working NSA backends ([msg 273]), the assistant established baseline throughput of approximately 225 output tokens per second ([msg 273]). However, profiling revealed a puzzling inefficiency: all eight GPUs were running at 100% utilization but drawing only 55% of their power budget (330W of 600W TDP) ([msg 268]). The GPUs were compute-bound but not saturating their tensor cores — a classic symptom of kernel launch overhead and small-matrix inefficiency in single-token decode.

The user had previously noted that "this is really new gpu" ([msg 280]) and suggested tuning the kernels. The assistant had already attempted several MoE runner backends: flashinfer_cutlass was the default, flashinfer_trtllm failed because it was SM100-only (datacenter Blackwell, not workstation SM120), and the prior Kimi K2 tuning work used Triton-based fused MoE kernels that were incompatible with the FP4 quantization path ([msg 275]). The assistant was running out of generic options.

The Discovery

Message [msg 283] opens with a moment of recognition: "There it is — gen_cutlass_fused_moe_sm120_module exists! FlashInfer has SM120-specific CUTLASS MoE." This exclamation reveals the significance — the assistant had been searching for SM120-specific code paths and finally found one. The discovery came from listing the contents of flashinfer.fused_moe and spotting the gen_cutlass_fused_moe_sm120_module function alongside its SM90 (Hopper) and SM100 (datacenter Blackwell) counterparts.

The assistant immediately probes deeper with two bash commands. The first calls help() on the SM120 module generator, revealing parameters like tune_max_num_tokens, min_latency_mode, and enable_pdl (Persistent Data Loader — a Blackwell CUTLASS feature). The second command searches for autotune and configuration files within the FlashInfer package, discovering autotuner.py and a collection of SM120-specific CUDA source files: gemm_groupwise_sm120.cu, group_gemm_sm120_binding.cu, fp4_gemm_cutlass_sm120.cu, and others.

Input Knowledge Required

To fully grasp this message, the reader needs several layers of context. First, an understanding of the hardware landscape: the RTX PRO 6000 Blackwell uses the SM120 architecture, distinct from the datacenter SM100 (B200/B100) and consumer SM120 is actually the workstation variant. Second, familiarity with the MoE execution model — that a transformer's feed-forward layers are replaced by multiple "expert" networks, with a router selecting which experts to activate per token, creating small matrix operations that are hard to parallelize efficiently. Third, knowledge of the software stack: FlashInfer is a kernel library for transformer inference, CUTLASS is NVIDIA's CUDA template library for matrix operations, and the "Persistent Data Loader" (PDL) is a Blackwell-specific feature for improving memory bandwidth utilization.

The reader must also understand the prior failure modes: flashinfer_trtllm MoE failed with an error because it was compiled only for SM100 ([msg 279]), and the Triton-based fused MoE kernels from the Kimi K2 work targeted FP8 quantization, not the NVFP4 format used by GLM-5. The assistant was in a narrowing search space, and this message represents the first expansion of that space.

The Thinking Process Visible

The reasoning in this message is structured as a classic scientific investigation. The assistant begins with a hypothesis ("FlashInfer has SM120-specific CUTLASS MoE"), then tests it with two parallel probes: examining the SM120 module generator's interface and searching for tuning infrastructure. The parallel structure of the bash commands (both dispatched in the same round) shows the assistant gathering complementary information simultaneously — one command reveals the API surface, the other reveals the implementation artifacts.

A subtle but important reasoning step is the connection between autotuner.py and the SM120 CUDA files. The assistant doesn't just find files; it implicitly connects the existence of an autotuner with the need to generate optimized kernel configurations for these new GPUs. The discovery of fp4_gemm_cutlass_sm120.cu is particularly significant because it confirms that the FP4 quantization path — the very format GLM-5 uses — has SM120-specific kernel implementations.

Assumptions and Potential Missteps

The assistant makes a key assumption: that the FlashInfer autotuner can be invoked to generate SM120-specific configurations that will outperform the generic defaults. This is reasonable but not guaranteed — the autotuner may not cover the specific matrix dimensions of GLM-5 (E=256, N=256 at TP8), or the generated configurations may not improve upon the existing flashinfer_cutlass backend. The assistant also assumes that the SM120-specific module (gen_cutlass_fused_moe_sm120_module) is the right path forward, without yet verifying that it supports the NVFP4 weight format used by the model.

Another assumption is that kernel tuning is the primary remaining bottleneck. The profiling data showed 100% GPU utilization at 55% power, which could also be explained by memory bandwidth saturation (52-53% memory utilization) or by the inherent arithmetic intensity limits of single-token decode through MoE layers. Tuning may improve kernel efficiency but cannot change the fundamental compute-to-communication ratio of the workload.

Output Knowledge Created

This message produces several concrete outputs. First, it establishes that FlashInfer contains SM120-specific CUTLASS MoE kernel generation code, with parameters for token batching (tune_max_num_tokens), latency optimization (min_latency_mode), and Blackwell-specific features (enable_pdl). Second, it maps the FlashInfer autotuning infrastructure, locating autotuner.py and the SM120 CUDA kernel sources. Third, it confirms the existence of FP4-specific SM120 kernels (fp4_gemm_cutlass_sm120.cu), validating that the quantization format is supported at the architecture level.

The message also creates negative knowledge: the assistant implicitly rules out the flashinfer_trtllm path (already failed), the Triton tuning path (incompatible quantization), and the generic flashinfer_cutlass defaults (suboptimal performance). The search space is now focused on FlashInfer's SM120 autotuning infrastructure.

Significance in the Larger Narrative

This message is the pivot point in the optimization arc. Before it, the assistant was trying pre-built backends and finding them inadequate. After it, the assistant will attempt to generate custom kernel configurations tuned for the specific GPU architecture and model dimensions. The discovery of gen_cutlass_fused_moe_sm120_module and the autotuner opens a new line of attack: instead of choosing among fixed backends, the assistant can now attempt to compile optimized kernels at runtime.

The message also demonstrates a key pattern in ML infrastructure debugging: when generic solutions fail, the path forward often involves exploiting architecture-specific code paths. The assistant's systematic search through FlashInfer's module hierarchy — from fused_moe to core to jit — mirrors the narrowing process of a debugger tracing through a call stack. Each failure eliminates a branch, until the remaining path is the architecture-specific one.

Conclusion

Message [msg 283] is a discovery message in the purest sense. No configuration is changed, no benchmark is run, no decision is finalized. But the knowledge it produces — that FlashInfer has SM120-specific CUTLASS MoE kernels with an autotuner — fundamentally reshapes the optimization strategy. The assistant moves from "which backend should I use?" to "how do I generate the optimal kernel configuration for this specific GPU and model?" This shift from selection to generation is the hallmark of deep performance optimization, and this message captures the moment that shift occurs.