Probing the Kernel Frontier: Investigating Autotune Mechanisms for Blackwell SM120 MoE Execution

Introduction

In the high-stakes world of large language model inference on novel hardware, every microsecond counts — and every kernel path is a potential bottleneck. Message 282 of this opencode session captures a pivotal moment of investigative exploration: after successfully deploying the GLM-5-NVFP4 model on 8× RTX PRO 6000 Blackwell GPUs and resolving a critical NaN crash during decode, the assistant turns its attention to the question of kernel tuning. The user's directive — "Also probaby tune the kernels, this is really new gpu" ([msg 280]) — sets the stage for a systematic probe into the FlashInfer library's autotune capabilities and MoE (Mixture-of-Experts) kernel generation functions. This message, though brief in its surface form, represents a deep and methodical reconnaissance of the software-hardware interface, seeking to understand what tuning levers exist for the SM120 GPU architecture before any tuning can begin.

The Context: A Working Deployment with Unsaturated Potential

To understand why this message matters, one must appreciate the journey that preceded it. The assistant had spent dozens of rounds battling NaN crashes, testing NSA decode backends, benchmarking throughput, and diagnosing virtualization-induced PCIe bottlenecks. By message 282, the deployment was working — the GLM-5-NVFP4 model was producing coherent output on 8 Blackwell GPUs with tensor parallelism 8. But the performance profile told a frustrating story: all 8 GPUs were running at 100% utilization yet drawing only 55% of their 600W power budget ([msg 268]). This asymmetry — fully occupied compute units operating at half power — strongly suggested that the GPU's tensor cores were not being saturated. The MoE kernels, responsible for routing tokens through the model's 256 experts, were likely executing with suboptimal block sizes and dispatch patterns for the novel SM120 architecture.

The user's intuition was correct: these RTX PRO 6000 Blackwell GPUs were new enough that pre-tuned kernel configurations might not exist in the FlashInfer library. The assistant had already discovered that the flashinfer_trtllm MoE backend was SM100-only (<msg id=279-281>), ruling out one potential optimization path. Now it needed to understand what was available for SM120 and whether FlashInfer exposed any autotune infrastructure.

The Message: A Two-Command Reconnaissance

The message executes two targeted Python probes on the remote server at 10.1.230.175. Both are executed via SSH, sourcing the ml-env virtual environment that contains the installed FlashInfer library.

Probe 1: Cataloging the API Surface

The first command imports flashinfer and prints the directory of flashinfer.fused_moe, filtering for keywords related to tuning, kernel backends, and FP4 quantization:

import flashinfer
print(dir(flashinfer.fused_moe))

The output reveals a rich set of functions organized by SM architecture version. Critically, the listing includes gen_cutlass_fused_moe_sm120_module — a function specifically designed to generate CUTLASS-based MoE kernel modules for the SM120 architecture. This is the assistant's first concrete evidence that FlashInfer has architecture-specific kernel generation, and that SM120 support exists. The listing also shows gen_cutlass_fused_moe_sm100_module, gen_cutlass_fused_moe_sm103_module, and gen_cutlass_fused_moe_sm90_module, confirming a pattern of per-architecture kernel generation. The presence of cutlass_fused_moe as a top-level function indicates the primary execution entry point, while gen_trtllm_gen_fused_moe_sm100_module confirms the SM100-only nature of the TRT-LLM path previously discovered.

Notably absent from the filtered output: any function explicitly named "autotune" or "tune." The tune keyword in the grep filter returned no matches, suggesting that FlashInfer's tuning mechanism — if it exists — is not exposed as a standalone function in this namespace. This is a significant finding: kernel tuning for these GPUs may require manual configuration or external tools rather than an automatic FlashInfer tuner.

Probe 2: Inspecting the Function Signature

The second command drills into the primary execution function:

from flashinfer.fused_moe import cutlass_fused_moe
help(cutlass_fused_moe)

The truncated output shows the function's parameter signature, revealing a complex interface designed for FP4 quantized MoE computation. The function accepts:

The Reasoning: Systematic Exploration of an Unknown Landscape

The assistant's thinking process in this message is a textbook example of systematic software reconnaissance. The reasoning unfolds in several layers:

Layer 1: The User's Directive as a Hypothesis — The user suggested kernel tuning ([msg 280]). The assistant accepts this as a working hypothesis but recognizes that tuning requires understanding what tools exist. You cannot tune what you cannot inspect.

Layer 2: Bounding the Search Space — The assistant already knows from the previous round ([msg 281]) that flashinfer_trtllm is SM100-only. This eliminates one backend. The remaining candidates are flashinfer_cutlass (currently in use), flashinfer_cutedsl (mentioned as an alternative), and any autotune infrastructure. The message explicitly frames the investigation: "Let me first check if flashinfer has any autotune mechanism we should enable, and look at the cutedsl path."

Layer 3: API Surface Probing — Rather than reading documentation or source code directly, the assistant probes the live Python environment. This is a pragmatic choice: documentation may be outdated or absent for a rapidly evolving library, but the installed code is the ground truth. The dir() call reveals the actual API surface available in the deployed environment.

Layer 4: Signature Analysis — Having found the relevant functions, the assistant inspects the cutlass_fused_moe signature to understand the data flow and identify potential tuning parameters. The complex parameter list hints at the many degrees of freedom in MoE kernel execution.

Assumptions and Their Implications

The message rests on several assumptions, some explicit and some implicit:

Assumption 1: FlashInfer is the Right Place to Look. The assistant assumes that kernel tuning for the MoE execution path is managed through FlashInfer rather than through SGLang's own configuration, NVIDIA's CUDA toolkit, or a separate tuning framework. This is a reasonable assumption given that FlashInfer provides the MoE kernel implementations, but it may miss tuning opportunities at higher or lower levels of the stack.

Assumption 2: Autotune Would Be Exposed as a Function. The search for "tune" in the function names assumes that autotune capabilities would be surfaced as callable Python functions. In practice, kernel autotuning in GPU libraries often happens at compile time (through template instantiation) or through separate command-line tools. The absence of a tune function does not mean tuning is impossible — it may mean tuning happens differently.

Assumption 3: The SM120 Architecture is Distinct Enough to Require Custom Tuning. This assumption is validated by the discovery of gen_cutlass_fused_moe_sm120_module, which confirms that FlashInfer treats SM120 as a distinct architecture requiring its own kernel generation path. The RTX PRO 6000 Blackwell (SM120) is indeed a different microarchitecture from datacenter Blackwell (SM100), with different tensor core capabilities and memory hierarchies.

Assumption 4: The Current Performance Gap is Kernel-Related. The assistant has previously established that GPUs are at 100% utilization but only 55% power ([msg 268]). The inference that this is due to suboptimal MoE kernels is plausible but not proven — it could also be caused by memory bandwidth limitations, pipeline stalls, or other architectural factors.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

GPU Architecture: Understanding that SM120 refers to the Blackwell architecture used in RTX PRO 6000 GPUs (consumer/workstation), distinct from SM100 (datacenter Blackwell like B200). The SM numbering scheme maps to NVIDIA's compute capability versions.

MoE Model Architecture: Knowledge that Mixture-of-Experts models route each token through a subset of expert networks, requiring dynamic dispatch and sparse matrix operations that are difficult to optimize for GPU execution. The GLM-5-NVFP4 model has 256 experts with specific hidden dimension sizes.

Quantization Formats: Understanding of NVFP4 (NVIDIA FP4) quantization, which packs two 4-bit weights into a single byte and requires specialized dequantization logic in the kernel. The modelopt_fp4 quantization path uses block-wise scaling factors.

FlashInfer Library: Familiarity with FlashInfer as a GPU kernel library for transformer inference, providing optimized implementations of attention, MoE, and other operations. The library uses architecture-specific code generation through CUTLASS templates.

SGLang Server Architecture: Understanding of the relationship between SGLang's server configuration flags (--moe-runner-backend, --nsa-decode-backend) and the underlying kernel implementations.

Output Knowledge Created

This message produces several concrete findings:

  1. Discovery of SM120-Specific Kernel Generation: The function gen_cutlass_fused_moe_sm120_module exists and is available in the installed FlashInfer version. This means architecture-specific kernel generation for SM120 is possible, though the function's exact usage and tuning parameters require further investigation.
  2. Confirmation of Architecture-Specific Code Paths: The pattern of gen_cutlass_fused_moe_sm{90,100,103,120}_module functions confirms that FlashInfer uses per-architecture kernel generation, with SM120 being one of the supported targets.
  3. No Obvious Autotune Function: The absence of a tune function in the filtered output suggests that automatic kernel tuning is not exposed as a simple API call. Tuning likely requires either manual configuration of kernel parameters or external tooling.
  4. Detailed Function Signature: The cutlass_fused_moe signature reveals the complexity of FP4 MoE execution, including the handling of quantization scales, expert routing, and activation functions. This signature provides the interface specification for any custom kernel tuning or replacement.
  5. Validation of the Investigation Direction: The discovery of SM120-specific code generation validates the user's intuition that these GPUs require special handling and that generic kernel configurations may be suboptimal.

The Broader Significance

This message exemplifies a critical phase in any ML infrastructure deployment: the transition from "making it work" to "making it fast." The NaN crashes have been resolved, the model produces correct output, and baseline throughput has been established. Now the hard work of optimization begins — and the first step of optimization is always measurement and discovery.

The assistant's methodical approach — probing the API surface, inspecting function signatures, and cataloging available tools — mirrors the workflow of a systems engineer encountering unfamiliar hardware. The discovery that SM120 has its own kernel generation function but no obvious autotune mechanism sets the stage for the next phase of work: either manually invoking the kernel generator with tuned parameters, exploring the cutedsl path, or developing a custom tuning pipeline.

In the broader narrative of this opencode session, message 282 marks the pivot point between debugging and optimization. The crisis of the NaN crash is behind them; the challenge of squeezing performance from novel hardware lies ahead. The assistant's systematic reconnaissance of the FlashInfer API surface is the foundation upon which all subsequent tuning efforts will be built — and the discovery that SM120 kernel generation exists, but autotune does not, will shape the optimization strategy for rounds to come.