The Autotuner That Didn't Tune: Diagnosing Runtime MoE Kernel Optimization on Blackwell GPUs

In the middle of an intense debugging session deploying the GLM-5-NVFP4 large language model on 8× NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs, the assistant produced a brief but pivotal message ([msg 290]) that crystallized an entire investigative thread spanning dozens of previous exchanges. The message reads:

The autotuner does runtime profiling when is_tuning_mode=True. SGLang sets this during warmup. Let me check what happens during our warmup: [bash] ssh 10.1.230.175 "grep -i 'autotun\|Autotuner\|tuning\|profiling\|tactic' ~/sglang-glm5.log | head -20" [2026-02-19 01:00:41] server_args=ServerArgs(model_path='lukealonso/GLM-5-NVFP4', tokenizer_path='lukealonso/GLM-5-NVFP4', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkp...

At first glance, this appears to be a simple check — run a grep command, look at the log. But the message represents a critical moment of synthesis: the assistant had spent the previous hour painstakingly reverse-engineering flashinfer's autotuner infrastructure, and this single command was the test that would confirm or refute its understanding of how the system actually behaved during server startup. The result — a single server_args line and nothing matching the grep patterns — told a story of its own.

The Context: A Long Trail of Investigation

To understand why this message matters, one must appreciate the investigative journey that preceded it. The session had been wrestling with a perplexing performance problem: the GLM-5-NVFP4 model was running on eight top-of-the-line Blackwell GPUs, yet the GPUs were showing only 55% power utilization despite 100% compute utilization. This suggested that the tensor cores were not being saturated — the MoE (Mixture-of-Experts) kernels were running but with poor efficiency, likely due to suboptimal tile configurations for the novel SM120 architecture.

The assistant had already explored multiple avenues. It had discovered that flashinfer's MoE runner had five distinct backends for FP4 quantization, including flashinfer_cutlass (the currently active one), flashinfer_trtllm (which crashed because it was SM100-only), and flashinfer_cutedsl. It had found that flashinfer contained dedicated SM120 code paths — fp4_gemm_cutlass_sm120.cu, gen_cutlass_fused_moe_sm120_module, and an entire autotuner.py module. Most importantly, it had discovered that flashinfer maintained per-device tuning configuration files at paths like flashinfer/tuning_configs/v0_1_trtllm_fused_moe_NVIDIA_B200.py — and that no such file existed for the RTX PRO 6000 Blackwell.

This was the root cause: flashinfer had pre-tuned configurations for datacenter Blackwell GPUs (B200, GB200) but not for the consumer/workstation RTX PRO 6000 (SM120). Without a tuning config, the load_from_file function returned (False, 0, -1, None) — meaning it fell back to default config 0 with tile_config -1, which was almost certainly suboptimal for this hardware.

The Key Insight: Runtime Autotuning

The assistant then dug into the autotuner source code. It discovered that flashinfer's AutoTuner class supported runtime profiling: when is_tuning_mode=True, it would profile different kernel configurations during actual execution and select the best one. The autotune context manager wrapped code blocks in tuning mode. And critically, SGLang's server warmup phase was supposed to enable this mode.

Message 290 is where this theoretical understanding meets empirical reality. The assistant states its conclusion clearly: "The autotuner does runtime profiling when is_tuning_mode=True. SGLang sets this during warmup." Then it runs the grep command to verify — to see if the server logs actually contain evidence of autotuning having occurred.

What the Output Revealed

The grep output is telling. It returns only a single server_args line — the server's startup configuration. There are no lines matching "autotun", "Autotuner", "tuning", "profiling", or "tactic". This is a significant non-result. It means one of three things:

The autotuner never entered tuning mode during warmup (perhaps the warmup path didn't trigger it for this model/quantization combination).
The autotuner ran but didn't produce log output at the default log level (the assistant had since restarted with --log-level debug in message 292, suggesting this was a concern).
The autotuner ran but found no configurations to tune (unlikely given the missing config file). The absence of autotuning log lines confirmed that the server had been running with default, untuned MoE kernels — directly explaining the poor GPU utilization observed earlier.

Assumptions and Reasoning

The assistant's reasoning in this message rests on several assumptions. First, it assumes that the autotuner's log output would include the specific keywords being grepped — a reasonable assumption given the source code inspection that revealed logger.info("[Autotuner]: Autotuning process starts ...") and similar messages. Second, it assumes that SGLang's warmup phase actually invokes the flashinfer autotuner for the flashinfer_cutlass MoE backend — an assumption that the negative result would call into question. Third, it assumes that the previous server log (from the flashinfer_trtllm crash attempt) is representative of the autotuner's behavior, when in fact that server never reached a functional state.

The message also reveals a subtle mistake in the assistant's mental model. It had been investigating the flashinfer_trtllm backend's log (which crashed), but the autotuner behavior for that backend might differ from the working flashinfer_cutlass backend. The assistant would realize this in the very next message ([msg 291]), where it notes: "The autotuning ran but for the failed flashinfer_trtllm attempt (that log is from the crash). Let me go back to the working flashinfer_cutlass config..."

Input and Output Knowledge

To fully understand this message, one needs knowledge of: flashinfer's autotuner architecture (the AutoTuner class, is_tuning_mode, the autotune context manager), SGLang's server warmup process, the relationship between tuning config files and runtime kernel selection, the SM120 architecture and its differences from SM100, and the specific MoE runner backends available for FP4 quantization.

The message creates new knowledge by establishing that the autotuner did not produce visible output during the server's startup. This negative result is itself valuable — it shifts the investigation from "are the tuned configs good?" to "is the autotuner even running?" It also validates the assistant's earlier discovery that no tuning config file exists for this GPU, since if runtime autotuning had compensated, there would be log evidence.

The Broader Significance

This message exemplifies a pattern that recurs throughout the session: the assistant builds an increasingly detailed mental model of a complex software system through source code inspection, then tests that model against empirical observation. The gap between theory and observation drives further investigation. Here, the theory said "autotuning should happen during warmup," but the observation said "no autotuning log output exists." This discrepancy would lead the assistant to restart the server with debug logging enabled ([msg 292]), to grep more specifically for autotuner activity ([msg 294]), and ultimately to consider whether the autotuner needed to be explicitly triggered or configured for this hardware.

In the broader narrative of the session, message 290 marks the transition from understanding the autotuner's architecture to diagnosing its actual behavior. It is the moment when the assistant stops reading source code and starts interrogating the running system — a shift from static analysis to dynamic analysis that is essential for debugging performance issues in complex distributed ML systems.