Peering Inside the Autotuner: The Quest for SM120 MoE Kernel Configs

Introduction

In the high-stakes world of deploying large language models on cutting-edge hardware, performance tuning often becomes a detective story. Message 289 of this opencode session captures a pivotal moment in such a story: the assistant, having successfully deployed the GLM-5-NVFP4 model on 8x NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs, turns its attention from what is running to how well it is running. The message is deceptively simple — a single bash command that SSHes into a remote machine and inspects the AutoTuner class within flashinfer's autotuner module. But beneath this technical veneer lies a critical juncture in the optimization process: the realization that the GPU architecture is so new that no pre-tuned kernel configurations exist, and the assistant must understand the autotuning infrastructure to generate them.

The Message

The subject message reads:

[assistant] [bash] ssh 10.1.230.175 'source ~/ml-env/bin/activate && python3 -c "
from flashinfer.autotuner import AutoTuner
import inspect
# Check the tune method
src = inspect.getsource(AutoTuner)
# Find tune-related methods
for line in src.split(chr(10)):
    if \"def \" in line or \"tune\" in line.lower() or \"best\" in line.lower() or \"config\" in line.lower():
        print(line)
" 2>&1'
class AutoTuner:
    """AutoTuner for optimizing TensorRT-LLM operations.
    different implementations and caching the best performing configurations.
    def __init__(self, warmup=3, repeat=10, stream_delay_micro_secs=1000):
        self.stats = AutoTunerStatistics()
    def get(cls):
            cls._instance = AutoTuner()
    def search_cache(
        tuning_config: TuningConfig,
        """Search for cached profiling results matching the current configuration.
            custom_op (str): T...

The command activates the Python virtual environment, imports the AutoTuner class from flashinfer's autotuner module, uses Python's inspect module to retrieve the class source code, and filters for lines containing keywords related to tuning, configuration, and best-practice selection. The output reveals the class skeleton: an __init__ method with warmup and repeat parameters, a singleton get method, and a search_cache method for looking up previously profiled configurations.

The Journey to This Point

To understand why this message was written, we must trace the path that led here. The session began with deploying GLM-5-NVFP4 — a massive mixture-of-experts (MoE) model with 256 experts — across eight RTX PRO 6000 Blackwell GPUs. After resolving a critical NaN crash during decode by selecting the correct NSA backends (trtllm), the assistant established baseline throughput metrics. However, a puzzling observation emerged: the GPUs were running at 100% utilization but only 55% power draw (330W out of 600W capacity). This indicated that the tensor cores were not being saturated — the kernels were running but inefficiently, likely limited by small matrix dimensions and per-expert dispatch overhead.

The user's comment at message 280 — "Also probaby tune the kernels, this is really new gpu" — crystallized the problem. The RTX PRO 6000 Blackwell (SM120 architecture) is a brand-new GPU. Unlike established architectures like NVIDIA's datacenter Blackwell (SM100, used in B200/GB200), the SM120 has no pre-existing tuning configurations in the software ecosystem. The assistant had already discovered in messages 285-288 that flashinfer's tuning configs directory contained only v0_1_trtllm_fused_moe_NVIDIA_B200.py and v0_1_trtllm_fused_moe_NVIDIA_GB200.py — nothing for the RTX PRO 6000. When a config file doesn't exist, the autotuner falls back to default configuration 0 with tile_config -1, which is almost certainly suboptimal for this architecture.

Why This Message Was Written

Message 289 represents a deliberate shift in strategy. Prior to this point, the assistant had been experimenting with different MoE runner backends — trying flashinfer_trtllm (which failed because it's SM100-only), considering flashinfer_cutedsl, and examining the flashinfer_cutlass path currently in use. But the user's directive to "tune the kernels" prompted a deeper question: how does the autotuner actually work, and what would it take to generate optimal configurations for SM120?

The assistant's reasoning, visible in the preceding messages, follows a logical progression:

1. Discovery: The autotuner looks for device-specific config files at flashinfer/tuning_configs/v0_1_trtllm_fused_moe_<device_name>.py (message 285).
2. Verification: No such file exists for the RTX PRO 6000 (message 285).
3. Understanding the fallback: When no config exists, load_from_file returns (False, 0, -1, None) — default config, no tile config (message 287).
4. Exploring the autotune mechanism: The autotune context manager enables tuning mode at runtime (message 288).
5. Inspecting the AutoTuner class: This is message 289 — examining the class internals to understand how to either run the tuner or manually construct configs. The motivation is clear: the assistant needs to understand the autotuner's API, its caching mechanism, and its configuration format to either (a) trigger a full autotuning run on the SM120 GPUs, or (b) manually construct a tuning config file by porting or adapting the B200 configs.

Input Knowledge Required

To fully grasp this message, several pieces of prior knowledge are essential:

The flashinfer autotuner architecture: The autotuner uses a singleton pattern (AutoTuner.get()) with a search_cache method that looks up previously profiled configurations by key. The tuning configs are stored as Python dictionaries mapping operation signatures to (config_id, tile_config_id) tuples. This is a CUTLASS-style tuning system where different tile sizes and kernel variants are benchmarked to find the fastest combination for each input shape.

The SM120 vs SM100 distinction: NVIDIA's Blackwell architecture has multiple variants. SM100 (used in B200/GB200 datacenter GPUs) and SM120 (used in RTX PRO 6000 workstation GPUs) have different CUTLASS tile configurations and kernel support. The flashinfer_trtllm MoE backend is SM100-only, while flashinfer_cutlass has separate module generators for each architecture: gen_cutlass_fused_moe_sm100_module, gen_cutlass_fused_moe_sm120_module, etc. (discovered in message 282).

The GLM-5-NVFP4 model dimensions: The model has 256 experts with hidden size 7168 and intermediate size 2048 (at TP8, each GPU handles N=256). These dimensions determine the matrix shapes that the MoE kernels must handle, which in turn determine which tile configurations are optimal.

The prior Kimi K2 tuning work: The user's repository contains MoE tuning configs generated by tuning_fused_moe_triton.py for a different model (GLM-4.7-FP8 with E=161, N=192/384). These are for the Triton MoE backend, not the CUTLASS backend, but they establish the precedent that hardware-specific tuning is necessary and effective.

Output Knowledge Created

This message produces several important pieces of knowledge:

1. AutoTuner class structure: The class has a __init__ with warmup=3, repeat=10, and stream_delay_micro_secs=1000 parameters — these control how many warmup iterations, measurement repetitions, and inter-measurement delays are used during profiling. The singleton pattern via get() ensures a single tuner instance. The search_cache method is the lookup mechanism for cached results.
2. The tuning infrastructure exists: Flashinfer does have an autotuning framework; it's not just static configs. The autotune context manager (seen in message 288) can enable tuning mode at runtime, which would profile different kernel variants and select the fastest.
3. The gap is in pre-computed configs: The autotuner can operate in two modes: using pre-computed configs from files, or running live profiling. The RTX PRO 6000 lacks the former but could potentially use the latter — though live profiling at runtime would add latency to the first few inference requests.
4. The config format is understood: The tuning configs map operation signatures (gemm_type, runner_type, input shapes, weight shapes) to (config_id, tile_config_id) pairs. This format is visible in the B200 config file examined in message 286.

The Thinking Process

The assistant's thinking process in this message is methodical and diagnostic. Having discovered that no SM120 tuning config exists, the assistant could have taken several approaches:

• Run the autotuner immediately: Trigger a full profiling run and wait for results.
• Copy the B200 config: Use the B200 config as a starting point, hoping SM120 compatibility.
• Inspect the code first: Understand the autotuner's internals before taking action. The assistant chose the third option, and the reasoning is sound. Running the autotuner without understanding its API could waste hours of GPU time. Copying the B200 config would likely produce suboptimal results because SM100 and SM120 have different CUTLASS tile configurations. By inspecting the AutoTuner class source, the assistant gains the knowledge needed to make an informed decision about how to proceed. The use of inspect.getsource(AutoTuner) with keyword filtering is a clever technique. Rather than reading the entire source file (which could be hundreds of lines), the assistant extracts only the lines relevant to tuning, configuration, and best-practice selection. This is a pattern-matching approach that efficiently surfaces the most important structural information. The output is truncated — the search_cache method signature is cut off mid-documentation — but enough information is visible to understand the class structure. The __init__ parameters (warmup=3, repeat=10, stream_delay=1000μs) are particularly valuable because they reveal the profiling methodology: three warmup runs to stabilize GPU clock speeds, ten measurement repetitions for statistical significance, and a 1ms delay between measurements to avoid thermal throttling interference.

Assumptions and Potential Mistakes

The message operates under several assumptions:

That the autotuner is the right tool for the job: The assistant assumes that flashinfer's CUTLASS-based MoE kernels are tunable via this autotuner and that generating optimal configs will meaningfully improve throughput. This is a reasonable assumption given that the B200 and GB200 have pre-computed configs, but it's possible that the SM120 kernels have different tuning parameters or that the performance bottleneck lies elsewhere (e.g., in the all-reduce communication rather than the MoE compute).

That the autotuner can be run on-demand: The assistant seems to assume that the autotuner can be triggered to profile all relevant kernel configurations. However, the autotune context manager (seen in message 288) only enables tuning mode — the actual profiling happens when kernels are first invoked. This means the autotuner profiles incrementally as different input shapes are encountered, which could take many inference requests to cover all configurations.

That the B200 config format is applicable: The assistant examined the B200 config file structure (message 286) and seems to assume that SM120 configs would follow the same format. While this is likely true (both use CUTLASS-based MoE), the actual tile configurations that are optimal for SM120 may differ significantly from SM100.

That the GPU is not thermally constrained: The 55% power utilization at 100% GPU utilization (observed earlier) suggests the kernels are not compute-bound. The assistant assumes this is due to suboptimal kernel configurations rather than, say, memory bandwidth limitations or PCIe bottlenecks. This assumption is being validated in parallel — the user also raised the possibility of virtualization-induced PCIe latency (mentioned in the chunk summary).

Broader Significance

Message 289 represents a microcosm of the challenges faced when deploying cutting-edge AI models on new hardware. The software ecosystem (flashinfer, sglang, PyTorch) evolves rapidly, but pre-tuned configurations always lag behind new GPU architectures. The assistant's methodical approach — discovering the gap, understanding the infrastructure, and planning the next steps — is a template for how to handle such situations.

The message also highlights the importance of understanding the tools you're working with. Rather than blindly running the autotuner or copying configs from a different architecture, the assistant took the time to inspect the code and understand the tuning mechanism. This investment in understanding pays dividends in the form of more targeted and effective optimization efforts.

In the broader narrative of this session, message 289 is the turning point where the assistant transitions from "making it work" to "making it fast." The NaN crash has been resolved, the model is generating coherent output, and now the focus shifts to squeezing every last drop of performance from these eight Blackwell GPUs. The autotuner inspection is the first step in that optimization journey — a journey that will ultimately lead to understanding the virtualization-induced PCIe bottlenecks that become the session's central performance insight.