Unearthing the Autotuner: A Pivotal Discovery in Kernel Optimization for Blackwell GPUs

In the complex ecosystem of large language model deployment on cutting-edge hardware, performance optimization is rarely a straight line. It is a process of iterative discovery, where each layer of abstraction peeled back reveals new questions. Message 284 in this opencode session represents one such revelatory moment — a point where the assistant, after a long chain of debugging and experimentation, turns from trying pre-configured kernel backends toward the deeper question of whether the kernels themselves can be tuned for the specific GPU architecture at hand. This message is not about a breakthrough in throughput; it is about the discovery of a tool that might enable one.

The Road to Message 284

To understand why this message was written, one must first understand the journey that led to it. The session had been wrestling with deploying the GLM-5-NVFP4 model — a large Mixture-of-Experts (MoE) language model with 256 experts using NVFP4 (NVIDIA FP4) quantization — across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The hardware was novel: these GPUs use the SM120 architecture, a consumer/workstation variant of Blackwell that differs significantly from the datacenter SM100 chips used in the B100/B200 series.

The team had already overcome a critical NaN crash during decode by selecting the correct attention backends (--nsa-decode-backend trtllm), established baseline throughput of approximately 225 output tokens per second with 64 concurrent requests, and diagnosed that the GPUs were running at 100% utilization but only drawing 55% of their 600W power budget — a telltale sign that the compute units were not being fully saturated. The bottleneck was not PCIe bandwidth or memory bandwidth, but rather the efficiency of the small-matrix operations inherent in single-token MoE decode.

Various MoE runner backends had been tested: flashinfer_cutlass was the default, flashinfer_trtllm was tried but failed because it is SM100-only, and flashinfer_cutedsl remained unexplored. Expert parallelism was evaluated and dismissed as unhelpful given the model's small hidden size and the PCIe-bound communication topology. The user, observing this struggle, offered a crucial insight in message 280: "Also probaby tune the kernels, this is really new gpu."

This suggestion — to tune rather than merely select — reframed the problem. The assistant had been treating the MoE kernels as fixed artifacts, choosing among pre-built backends. But the user recognized that the RTX PRO 6000 Blackwell was sufficiently new that the kernel configurations shipped with flashinfer might be suboptimal for its specific SM120 architecture. Tuning, rather than selection, was the path forward.

The Message: Discovery of the Autotuner

In message 284, the assistant acts on this insight. The message begins with the assistant's observation:

Interesting — flashinfer has dedicated SM120 code for FP4 GEMM (fp4_gemm_cutlass_sm120.cu, fp4_gemm_template_sm120.h), SM120-specific CUTLASS collective builders, and an autotuner.py. Let me look at the autotuner and the GEMM config options:

This opening line is itself a synthesis of the discoveries made in the immediately preceding messages (282 and 283). In those messages, the assistant had probed the flashinfer library and found:

import flashinfer.autotuner as at
print(dir(at))

The output reveals a rich set of classes and functions: AutoTuner, AutoTunerStatistics, TuningConfig, TunableRunner, OptimizationProfile, ConstraintSpec, DynamicDim, StaticDim, autotune, get_config_path, and many others. This is not a trivial utility — it is a full autotuning framework embedded within flashinfer.

The second command reads the first 100 lines of autotuner.py itself, showing imports from Python standard library (contextlib, copy, importlib, inspect, itertools, os), from abc (abstract base classes), from dataclasses, and from torch. Notably, it imports delay_kernel from flashinfer.tllm_utils (with a commented-out alternative from tensorrt_llm.bindings), and uses from .jit.core import logger — indicating this autotuner is part of flashinfer's JIT compilation system.

Reasoning and Motivation

The assistant's reasoning in this message is methodical and hypothesis-driven. The chain of logic proceeds as follows:

  1. Observation: The GPUs are at 100% utilization but only 55% power. The tensor cores are not being saturated by the current MoE kernels.
  2. Hypothesis: The default kernel configurations shipped with flashinfer are not optimal for the SM120 architecture. The RTX PRO 6000 Blackwell is a new GPU (released in 2025), and pre-tuned configurations may not exist for it.
  3. Evidence gathering: In messages 282 and 283, the assistant discovered that flashinfer does have SM120-specific code paths — gen_cutlass_fused_moe_sm120_module, fp4_gemm_cutlass_sm120.cu, etc. This confirms that the library developers anticipated SM120 support, but it doesn't guarantee that the default configurations are tuned.
  4. New lead: The presence of autotuner.py suggests there is a mechanism to generate tuned configurations, not just use pre-baked ones.
  5. Action: Investigate the autotuner to understand its interface, capabilities, and whether it can be applied to the FP4 MoE kernels on SM120. The motivation is clear: the assistant has exhausted the strategy of trying different pre-built MoE runner backends. The flashinfer_trtllm path failed on SM120. The flashinfer_cutlass path works but delivers suboptimal performance. The flashinfer_cutedsl path is untested. The user's suggestion to tune the kernels opens a new dimension of optimization — instead of choosing among fixed configurations, generate configurations tailored to this specific GPU-model combination.

Assumptions Embedded in This Message

Every investigation rests on assumptions, and this message is no exception. Several implicit assumptions are at work:

That the autotuner is applicable to the FP4 MoE path. The assistant discovered autotuner.py in the flashinfer package, but it is not yet confirmed that this autotuner can generate configurations for the cutlass_fused_moe function with NVFP4 quantization on SM120. The autotuner might be designed for other components — attention kernels, MLA (Multi-head Latent Attention) kernels, or GEMM operations in different quantization formats.

That tuning will improve throughput. The assistant assumes that the current suboptimal performance is at least partly due to poor kernel configuration choices (block sizes, tile dimensions, thread layouts) rather than fundamental architectural limitations of the SM120 GPU or the PCIe communication topology. If the bottleneck is truly the small matrix sizes inherent in single-token MoE decode (where each token activates only a few experts, each with a small hidden dimension), then even perfectly tuned kernels may not dramatically improve throughput.

That the autotuner can be invoked from the command line or integrated into the sglang server startup. The assistant has not yet explored how to actually run the autotuner. The dir() output shows an autotune function and a get_config_path function, but the workflow for generating, persisting, and loading tuned configurations is unknown.

That the remote machine has sufficient resources to run the autotuner. Kernel autotuning typically involves running many candidate configurations and measuring their performance. On a machine with 8 GPUs already loaded with a running model, this could be resource-intensive or conflict with the server process.

Input Knowledge Required

To fully grasp this message, a reader needs substantial domain knowledge spanning several layers of the ML inference stack:

GPU Architecture: Understanding that SM120 is the compute architecture of the RTX PRO 6000 Blackwell, a workstation GPU, distinct from the SM100 architecture used in datacenter Blackwell GPUs (B100/B200). The SM120 has different characteristics — different tensor core generations, different cache hierarchies, different optimal kernel parameters.

Quantization Formats: NVFP4 (NVIDIA FP4) is a 4-bit floating-point quantization format used to compress model weights. The GLM-5 model uses this format for its MoE expert weights. FP4 GEMM (General Matrix Multiply) kernels are significantly more complex than FP16 or FP8 kernels because they must handle the packing and unpacking of 4-bit values.

MoE Architecture: Mixture-of-Experts models have multiple "expert" sub-networks, with a router selecting which experts to activate for each token. The GLM-5 model has 256 experts, and in decode mode, typically only a small number (top-2 or top-3) are activated per token, resulting in small matrix multiplications that are hard to parallelize efficiently.

The FlashInfer Library: FlashInfer is a GPU kernel library for LLM inference, providing optimized implementations of attention, MoE, and other operations. It uses a JIT (Just-In-Time) compilation system to generate kernels at runtime, and it has an autotuning framework for exploring kernel configurations.

CUTLASS: A CUDA template library for GEMM operations developed by NVIDIA. The cutlass_fused_moe function in flashinfer uses CUTLASS templates to generate fused MoE kernels. The enable_pdl parameter refers to Persistent Data Loader, a CUTLASS feature for Blackwell GPUs that improves data loading efficiency.

Autotuning Concepts: Understanding that kernel autotuning involves defining a search space of parameters (tile sizes, thread counts, loop unrolling factors, etc.), running benchmarks for each configuration, and selecting the best one. The TuningConfig, OptimizationProfile, and TunableRunner classes in the autotuner output correspond to these concepts.

Output Knowledge Created

This message creates several pieces of new knowledge:

Confirmation of the autotuner's existence and structure: The dir() output reveals the full API surface of flashinfer.autotuner. Key classes include AutoTuner (the main tuner class), AutoTunerStatistics (for tracking tuning results), TuningConfig (representing a kernel configuration), TunableRunner (a kernel that can be tuned), OptimizationProfile (a set of tuning objectives), ConstraintSpec (constraints on valid configurations), and Dim, DynamicDim, StaticDim (for specifying kernel dimension parameters). The presence of get_config_path suggests a mechanism for persisting and loading tuned configurations.

Evidence of the autotuner's integration with flashinfer's JIT system: The import of delay_kernel from flashinfer.tllm_utils and the use of .jit.core.logger indicate that the autotuner is designed to work with flashinfer's JIT compilation pipeline — generating and compiling kernel variants on the fly.

A roadmap for the next investigation steps: The assistant now knows what to explore next — how to invoke the autotuner, what parameters it accepts, whether it supports the FP4 MoE path on SM120, and how to integrate tuned configurations into the sglang server.

Documentation of the SM120-specific code paths: The message (and its predecessors) establish that flashinfer has dedicated SM120 support for FP4 GEMM, including fp4_gemm_cutlass_sm120.cu, fp4_gemm_template_sm120.h, and gen_cutlass_fused_moe_sm120_module. This is valuable knowledge for anyone deploying models on RTX PRO 6000 Blackwell GPUs.

The Thinking Process Visible in the Message

The assistant's thinking process is visible in several dimensions:

The structure of the investigation is systematic: The assistant first discovers the autotuner's existence (msg 283), then probes its API (msg 284, first command), then reads its source code (msg 284, second command). This is a classic "discover → inspect → understand" pattern.

The assistant is building a mental model of the autotuner's architecture: The dir() output is not just a list of names — it reveals relationships. The presence of both AutoTuner (the orchestrator) and TunableRunner (the tunable kernel) suggests a two-level architecture where the tuner manages multiple runners. The OptimizationProfile class indicates that tuning can optimize for different objectives (latency, throughput, memory). The ConstraintSpec class suggests that not all parameter combinations are valid — there are hardware or algorithm constraints.

The assistant is looking for integration points: The get_config_path function is particularly interesting — it suggests that tuned configurations are saved to a file and can be loaded later. This is crucial for the deployment scenario: the assistant needs to know whether tuned configurations can be generated once and then used automatically by the sglang server on subsequent launches.

There is an implicit prioritization: The assistant chooses to investigate the autotuner before trying the remaining untested backends (flashinfer_cutedsl, cutlass from sgl_kernel). This is a strategic decision — if the autotuner can generate optimal configurations for the current flashinfer_cutlass backend, that might be more impactful than trying other backends that may also be untuned.

Limitations and Unanswered Questions

While this message represents progress, it also reveals gaps that remain to be filled:

The autotuner's specific applicability to FP4 MoE is not confirmed. The dir() output shows general-purpose tuning infrastructure, but the assistant has not yet verified that the autotuner supports the cutlass_fused_moe function with NVFP4 quantization. The autotuner might be designed for different kernel types.

The workflow for using the autotuner is unknown. How is the autotuner invoked? Does it require a separate script, or can it be triggered from within the sglang server? How long does tuning take? Does it need to run on idle GPUs, or can it run concurrently with inference?

The relationship between the autotuner and the existing SM120 code paths is unclear. Does gen_cutlass_fused_moe_sm120_module already use the autotuner internally? Or is the autotuner a separate system that generates configurations for the module generator?

The impact of tuning on the specific bottleneck is uncertain. Even with perfectly tuned kernels, the fundamental issue of small-batch MoE decode (1 token × small hidden dimension per expert) may limit the gains achievable through kernel optimization alone.

Significance in the Broader Narrative

Message 284 sits at a critical juncture in the session. The assistant has moved through several phases:

  1. Environment setup: Installing drivers, CUDA, and dependencies
  2. Crash debugging: Resolving the NaN decode crash through NSA backend selection
  3. Baseline measurement: Establishing throughput metrics
  4. Parameter tuning: Adjusting mem-fraction, CUDA graphs, and MoE backends
  5. Bottleneck diagnosis: Identifying compute-bound but power-limited execution
  6. Kernel exploration (current phase): Investigating whether kernel tuning can improve efficiency This message marks the transition from phase 5 to phase 6. The assistant has accepted the user's suggestion that kernel tuning is the next frontier, and has begun the technical investigation needed to make it happen. The discovery of the autotuner is the first step in what could be a significant optimization — or a dead end that forces a return to other strategies. The message also demonstrates a key characteristic of effective ML engineering: the willingness to dig deeper into the software stack. Rather than accepting the flashinfer library as a black box with fixed backends, the assistant explores its internals, discovers the autotuner, and begins to understand how it might be leveraged. This is the kind of systems-level thinking that separates deployment that merely works from deployment that performs.

Conclusion

Message 284 is a moment of discovery and redirection. The assistant, guided by the user's insight that kernel tuning is necessary for this new GPU architecture, uncovers the autotuner framework embedded within flashinfer. The message captures the beginning of a new investigative thread — one that moves from selecting among pre-built kernel backends toward generating custom-optimized configurations for the specific hardware-model combination.

The message is rich with implicit knowledge about GPU architecture, quantization formats, MoE inference, and kernel optimization. It reveals a systematic thinking process that proceeds from observation to hypothesis to evidence gathering to action. And it leaves the reader with a clear sense of what comes next: understanding the autotuner's interface, running it for the FP4 MoE path on SM120, and integrating the results into the deployment pipeline.

In the broader arc of the session, this message represents the moment when the team stopped asking "which backend should we use?" and started asking "how do we build the optimal backend for our hardware?" — a subtle but crucial shift in mindset that often separates adequate deployments from exceptional ones.