Unlocking Blackwell Performance: The Hunt for MoE Kernel Tuning Configs in FlashInfer

Introduction

In the high-stakes world of large language model deployment on cutting-edge hardware, the difference between acceptable performance and peak throughput often comes down to a single missing file. This article examines a pivotal moment in an opencode coding session where an AI assistant, deep in the trenches of deploying the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, uncovers the inner workings of FlashInfer's autotuning system. The message at index 287 represents a critical juncture where raw investigation meets systematic reverse-engineering—a moment when the assistant transitions from observing that "something is missing" to understanding exactly how the missing piece fits into the larger machinery.

The Context: A Performance Mystery Unfolds

The session leading up to this message had been a marathon of debugging and optimization. The team had successfully deployed the GLM-5-NVFP4 model using SGLang on eight RTX PRO 6000 Blackwell GPUs (SM120 architecture), but performance was puzzling. GPU utilization was pegged at 100% during decode, yet power draw hovered at only 55% of the 600W TDP. This discrepancy suggested that the GPU's tensor cores were not being fully saturated—the kernels were running, but inefficiently.

Earlier investigation revealed that FlashInfer, the kernel library powering SGLang's MoE (Mixture-of-Experts) operations, had pre-tuned configuration files for NVIDIA's datacenter Blackwell GPUs (B200 and GB200, both SM100 architecture) but no configuration file for the RTX PRO 6000 (SM120). This was discovered in message 285, where the assistant checked for the existence of the tuning config file at the path flashinfer.tuning_configs.v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition and found it did not exist. The only configs present were for NVIDIA_B200 and NVIDIA_GB200.

In message 286, the assistant examined the B200 config file to understand its format. The config was a Python dictionary mapping tuples of (gemm_type, runner_type, (batch_size, hidden_size), weight_shape, ...) to (config_id, tile_config_id) pairs. These tile configuration IDs select specific CUTLASS kernel variants—optimized for particular matrix dimensions and hardware characteristics. The B200 configs were useless for SM120 because CUTLASS tile configurations differ between architectures.

The Subject Message: Dissecting the Autotuner

Message 287 opens with the assistant synthesizing its understanding of the tuning config format:

The tuning config maps (gemm_type, runner_type, (batch_size, hidden_size), weight_shape, ...) to (config_id, tile_config_id). These are CUTLASS tile configs that differ between SM100 (B200) and SM120 (RTX PRO 6000).

This brief paragraph encapsulates a crucial insight. The assistant has recognized that the configs are not just arbitrary parameters—they are CUTLASS tile configurations, which are inherently architecture-specific. CUTLASS (CUDA Templates for Linear Algebra Subroutines) generates optimized matrix multiply kernels by selecting tile sizes for thread blocks, warps, and individual threads. These tile sizes must be tuned for each GPU architecture's register file size, shared memory capacity, and compute unit layout. SM100 (datacenter Blackwell) and SM120 (workstation Blackwell) have different hardware characteristics, so B200 configs cannot simply be reused.

The assistant then pivots to the next logical question: how does FlashInfer behave when no config file exists? Does it fall back to generic defaults, or does it attempt runtime autotuning? The key flag is --disable-flashinfer-autotune, which the assistant notes is currently set to False—meaning autotuning is enabled in the current server configuration, but what does that actually do when no config file is present?

To answer this, the assistant executes a bash command that reads the source code of the load_from_file function from FlashInfer's autotuner module:

@lru_cache(maxsize=None)
def load_from_file(key):
    module_name = get_config_path(is_module=True)
    try:
        module = importlib.import_module(module_name)
        best_configs = module.best_configs
    except (ImportError, AttributeError):
        best_configs = None
    if best_configs is not None:
        k = str((key[0], key[1], key[3]))
        if k in best_configs:
            logger.info(f"[Autotuner]: Loading configs for {k} from file.")
            return True, best_configs[k][0]...

The output is truncated in the conversation, but the critical logic is visible. The function attempts to import the device-specific module (e.g., flashinfer.tuning_configs.v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition). If the import fails (which it will, since the file doesn't exist) or if the module lacks a best_configs attribute, it sets best_configs = None. When best_configs is None, the function presumably falls through to some default behavior—either returning a fallback configuration or triggering runtime autotuning.

The Reasoning Process: Systematic Debugging in Action

What makes this message particularly instructive is the clear chain of reasoning it reveals. The assistant is not randomly probing; it is following a methodical investigation:

Observation: GPU utilization is 100% but power draw is only 55% of TDP. This indicates suboptimal kernel efficiency.
Hypothesis: The MoE kernels are not tuned for this GPU architecture.
Evidence gathering: Checked FlashInfer's tuning config directory—no config exists for RTX PRO 6000 (SM120), only for B200/GB200 (SM100).
Understanding the format: Examined the B200 config to learn the mapping structure.
Verification: Confirmed that these are CUTLASS tile configs, which are architecture-specific and cannot be shared between SM100 and SM120.
Next step: Investigate what happens at runtime when no config exists—does the autotuner kick in? This progression from observation → hypothesis → evidence → understanding → next action is a textbook example of systematic debugging. Each step is grounded in concrete data, and each conclusion leads naturally to the next investigation.

Assumptions and Their Implications

Several assumptions underpin this message. The most significant is that CUTLASS tile configurations genuinely differ between SM100 and SM120. This is almost certainly correct—NVIDIA's SM architectures evolve with each generation, and even within the same generation (Blackwell), the datacenter and workstation variants have different hardware resources. SM120 has different register counts, shared memory sizes, and warp scheduling characteristics than SM100. However, the assistant does not verify this assumption by, say, comparing the SM100 and SM120 CUTLASS template parameters. It accepts it as a given, which is reasonable given NVIDIA's history of architecture-specific tuning.

Another assumption is that having a tuned config file will materially improve performance. This is likely true—the B200 and GB200 configs exist precisely because the FlashInfer developers found that autotuning produced better results than generic defaults. But the magnitude of improvement is unknown. It could be a 5% gain or a 50% gain. The assistant is operating under the reasonable belief that this is a worthwhile optimization path.

A third assumption, visible in the assistant's choice to inspect load_from_file, is that understanding the autotuner's fallback behavior will inform the next steps. This is correct—if the autotuner silently falls back to reasonable defaults, the priority might shift to other optimization avenues. If it falls back to terrible defaults or crashes, generating a config becomes urgent.

Input Knowledge Required

To fully understand this message, the reader needs familiarity with several domains:

Mixture-of-Experts (MoE) architectures: The GLM-5-NVFP4 model uses MoE layers where each token is routed to a subset of "expert" sub-networks. Efficient MoE execution requires specialized kernels that can handle the irregular, sparse computation patterns.
CUTLASS and tile-based GEMM: CUTLASS is a CUDA template library for matrix multiplication that decomposes operations into hierarchical tiles. The choice of tile sizes profoundly impacts performance and must be tuned for each GPU architecture.
FlashInfer: A GPU kernel library for transformer inference that provides optimized implementations of attention, MoE, and other operations. It includes an autotuning framework for selecting optimal kernel configurations.
NVIDIA Blackwell architecture: The RTX PRO 6000 uses the SM120 variant of Blackwell, while datacenter GPUs like B200 use SM100. These have different hardware characteristics despite sharing the Blackwell name.
SGLang server architecture: Understanding that SGLang uses FlashInfer for MoE operations and exposes flags like --disable-flashinfer-autotune to control autotuning behavior.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

Confirmation of the config loading mechanism: The load_from_file function uses Python's importlib to dynamically load a device-specific module. If the module doesn't exist or lacks best_configs, it returns None (or a fallback).
The config key structure: The mapping uses a tuple of (gemm_type, runner_type, (batch_size, hidden_size), weight_shape, ...) as the key. This tells us that tuning is sensitive to both the GEMM operation type (gemm1 vs gemm2 for different parts of the MoE computation) and the specific matrix dimensions encountered at runtime.
The role of --disable-flashinfer-autotune: The assistant notes this flag is currently False, meaning autotuning is enabled. But the investigation reveals that "autotuning enabled" doesn't mean runtime autotuning is happening—it means the autotuner will try to load a config file, and if none exists, it falls back to defaults.
A roadmap for the next steps: The investigation naturally points toward either (a) generating a tuning config by running FlashInfer's autotuner on the RTX PRO 6000 hardware, or (b) manually creating config entries based on the model's specific dimensions.

The Broader Significance

This message exemplifies a pattern that recurs throughout ML infrastructure work: the gap between "it works" and "it works well" is often filled by hardware-specific tuning that hasn't been done yet. The RTX PRO 6000 Blackwell was a new GPU at the time of this session, and FlashInfer's developers had prioritized tuning for datacenter Blackwell variants (B200, GB200) over the workstation variant. This is a natural consequence of the ML ecosystem's focus on datacenter hardware, but it leaves early adopters of workstation Blackwell GPUs in a position where they must either wait for upstream tuning or do it themselves.

The assistant's methodical approach—discovering the missing config, understanding its format, tracing the autotuner's fallback behavior—provides a template for anyone facing similar "new hardware, no tuning" situations. Rather than guessing at kernel parameters or applying generic optimizations, the assistant goes straight to the source code to understand exactly how the system behaves when configuration is absent. This is the difference between cargo-cult optimization and informed engineering.

Conclusion

Message 287 represents a turning point in the optimization effort. The assistant has moved from observing a performance anomaly to understanding the root cause (missing SM120-specific CUTLASS tile configs) and tracing the mechanism by which FlashInfer handles this absence. The next steps are clear: either generate a tuning config by running FlashInfer's autotuner on the target hardware, or manually construct config entries based on the model's dimensions. Either path requires the foundational understanding established in this message—that the configs are architecture-specific CUTLASS tile selections, that the loading mechanism uses Python module import, and that the fallback behavior when no config exists is to use generic defaults.

In the broader narrative of the coding session, this message is where the assistant stops asking "what's wrong?" and starts asking "how do we fix it?"—armed with precise knowledge of the system's internals rather than vague hypotheses. It's a masterclass in systematic debugging and a reminder that in ML infrastructure, performance is often hiding in the details of kernel configuration.