The Autotuner's Missing Config: Debugging MoE Kernel Performance on a New Blackwell GPU

Introduction

In the complex ecosystem of deploying large language models on cutting-edge hardware, few problems are as frustrating as the "it works, but slowly" scenario. Message 292 of this opencode session captures a pivotal moment in precisely such a situation: the assistant is deploying GLM-5-NVFP4—a 256-expert mixture-of-experts model with NVFP4 quantization—across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The model runs without crashes, but GPU utilization tells a troubling story: 100% utilization at only 55% power draw (330W out of 600W). The tensor cores are not being saturated. The kernels run, but inefficiently.

This message represents a deliberate investigative pivot. Having resolved the critical NaN crash during decode (by selecting trtllm NSA backends) and established baseline throughput metrics (~225 output tokens/second), the assistant now turns to a deeper performance question: are the MoE kernels using optimal tile configurations for this specific GPU architecture? The answer, uncovered through a chain of investigations in preceding messages, is a definitive "no"—and message 292 is the action taken to address it.

The Full Message

The subject message reads as follows:

Let me restart with the working config and also set FLASHINFER_AUTOTUNER_LOAD_FROM_FILE=1 to force the autotune to try harder. But first — the autotuner only profiles when autotuning is enabled AND during warmup. I noticed disable_flashinfer_autotune=False in the config, so autotuning should run. Let me restart and capture the autotuning output:

>

``bash ssh -f 10.1.230.175 'source ~/ml-env/bin/activate && export NCCL_IB_DISABLE=1 && export NCCL_P2P_LEVEL=PHB && export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 && export NCCL_MIN_NCHANNELS=8 && export OMP_NUM_THREADS=8 && export SAFETENSORS_FAST_GPU=1 && export CUDA_HOME=/usr/local/cuda-12.8 && nohup python3 -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 --reasoning-parser glm45 --tool-call-parser glm47 --trust-remote-code --tp 8 --mem-fraction-static 0.92 --max-running-requests 64 --kv-cache-dtype auto --quantization modelopt_fp4 --attention-backend flashinfer --fp8-gemm-backend cutlass --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutlass --disable-custom-all-reduce --enable-flashinfer-allreduce-fusion --host 0.0.0.0 --port 8000 --log-level debug > ~/sglang-glm5.log 2>&1 &' ``

The Reasoning: Why This Message Was Written

To understand why this message exists, one must trace the investigative thread that led to it. The session had already spent considerable effort getting GLM-5-NVFP4 to run at all. The model uses a novel NSA (Native Sparse Attention) mechanism that, on the SM120 architecture of the RTX PRO 6000, initially produced NaN values during decode. This was resolved by explicitly selecting --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm, bypassing the default backends that were incompatible with these GPUs.

Once the model produced coherent output, the focus shifted to performance. The assistant had explored alternative MoE runner backends: flashinfer_trtllm crashed because it is SM100-only (datacenter Blackwell), while flashinfer_cutlass worked but left performance on the table. The key discovery came in messages 285–287, where the assistant probed flashinfer's autotuner infrastructure and found that no tuning configuration exists for the RTX PRO 6000 Blackwell (SM120). The tuning_configs/ directory contained only v0_1_trtllm_fused_moe_NVIDIA_B200.py and v0_1_trtllm_fused_moe_NVIDIA_GB200.py—configs for datacenter Blackwell GPUs, not the consumer/workstation SM120 variant.

This is a critical gap. The CUTLASS-based MoE kernels used by flashinfer_cutlass rely on pre-tuned tile configurations—choices about how to partition matrix multiplications across tensor cores—to achieve optimal throughput. Without a device-specific config, the autotuner falls back to (False, 0, -1, None): default config 0 with no tile configuration override. For a GPU architecture as new as Blackwell SM120, these defaults are unlikely to be optimal.

Message 292 is the assistant's response to this discovery. It represents a deliberate attempt to force the autotuner to do its job and capture the results.

Decisions Made in This Message

Several decisions are visible in this message, each reflecting a strategic choice about how to proceed.

Decision 1: Stick with flashinfer_cutlass. The assistant could have continued exploring alternative MoE backends—flashinfer_cutedsl (CuteDSL grouped GEMM) or cutlass via sgl_kernel were mentioned as possibilities in earlier messages. Instead, the decision is to optimize the currently working path rather than chase alternatives. This is pragmatic: flashinfer_cutlass works, and the problem is likely suboptimal tile configurations, not a fundamentally wrong kernel choice.

Decision 2: Set FLASHINFER_AUTOTUNER_LOAD_FROM_FILE=1. This environment variable is the assistant's attempt to "force the autotune to try harder." The variable name suggests it controls whether the autotuner loads configurations from a file (as opposed to, perhaps, using built-in defaults). By setting it to 1, the assistant hopes to trigger a more aggressive autotuning pass. However, the exact semantics of this flag are unclear from the investigation—the assistant is experimenting.

Decision 3: Enable debug logging. The addition of --log-level debug is a deliberate instrumentation choice. The assistant wants to capture the autotuner's output during warmup to understand what configurations are being tried, what falls back to defaults, and whether any profiling actually occurs. This is essential diagnostic information that was missing from previous runs.

Decision 4: Preserve the full working configuration. The bash command carefully reproduces every flag from the previously working setup: the NCCL environment variables for PCIe-bound communication (NCCL_P2P_LEVEL=PHB, NCCL_IB_DISABLE=1), the CUDA graphs and memory fraction settings, the NSA backend selection, and the flashinfer_cutlass MoE runner. Nothing is changed except the addition of debug logging. This minimizes confounding variables.

Assumptions Made

The message rests on several assumptions, some explicit and some implicit.

Assumption 1: The autotuner will actually run. The assistant notes that "disable_flashinfer_autotune=False in the config, so autotuning should run." This assumes that the autotuner is triggered during server warmup when the flag is not disabled. However, the autotuner's behavior may depend on more than just this flag—it may require specific conditions (e.g., a minimum number of warmup iterations, or the presence of representative input shapes) to activate.

Assumption 2: The autotuner can find better configurations. This is the core assumption driving the entire action. The assistant believes that the RTX PRO 6000's SM120 architecture is sufficiently different from B200/GB200 (SM100) that the default tile configurations are suboptimal, and that runtime profiling can discover better ones. This is plausible but unproven—it's possible that the default configs are already near-optimal for this model's dimensions.

Assumption 3: FLASHINFER_AUTOTUNER_LOAD_FROM_FILE=1 will help. This environment variable's exact effect is unknown. The assistant is essentially trying a flag and hoping it improves autotuning behavior. It may have no effect, or it may change the autotuner's behavior in unexpected ways (e.g., by loading a non-existent file and erroring out).

Assumption 4: Debug logging won't significantly impact performance or startup time. Adding --log-level debug increases logging volume, which could theoretically affect warmup timing or mask autotuner output in noise. The assistant assumes the benefit of visibility outweighs these risks.

Potential Mistakes and Incorrect Assumptions

The most significant potential mistake is the assumption that the autotuner will actually produce better configurations at runtime. Flashinfer's autotuner, as investigated in messages 287–289, works by profiling different kernel configurations during warmup and selecting the fastest. However, this profiling itself takes time and may not converge to optimal configurations within the warmup window. The autotuner's search_cache method suggests it caches results, but if the warmup is too short or the input shapes don't match production usage, the tuned configs may be suboptimal.

Another subtle issue: the autotuner's tuning configs map specific operation signatures (gemm type, runner type, batch size, hidden size, weight shapes) to tile config IDs. The GLM-5-NVFP4 model has specific dimensions (E=256 experts, N=256 intermediate size at TP8) that may not match any profiled configuration. If the autotuner only profiles for the shapes seen during warmup (which may be small, synthetic inputs), the tuned configs may not generalize to production batch sizes.

There's also a potential misunderstanding about FLASHINFER_AUTOTUNER_LOAD_FROM_FILE. The name suggests it controls whether to load from a file—but if the file doesn't exist (which we know it doesn't for RTX PRO 6000), setting this flag to 1 might cause an error or be silently ignored. The assistant may be conflating "load from file" with "run autotuning."

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains:

GPU Architecture: Understanding that Blackwell comes in multiple variants (SM100 for datacenter B200/GB200, SM120 for RTX PRO 6000) and that these have different optimal kernel configurations. The distinction between "works" and "works efficiently" on tensor core operations.

MoE Model Architecture: Knowledge that GLM-5-NVFP4 is a mixture-of-experts model with 256 experts, using NVFP4 (NVIDIA FP4) quantization. Understanding that MoE inference involves routing tokens to experts, computing GEMM operations on expert weights, and combining results—all of which benefit from kernel fusion and optimal tile sizing.

SGLang Server Configuration: Familiarity with the extensive set of flags controlling attention backends, MoE runners, quantization paths, and communication strategies. Understanding that --moe-runner-backend flashinfer_cutlass selects a specific CUTLASS-based fused MoE kernel implementation.

Flashinfer's Autotuner Infrastructure: Knowledge that flashinfer includes a runtime autotuner that profiles kernel configurations and caches results, that it looks for device-specific config files in tuning_configs/, and that the absence of such a file for the RTX PRO 6000 means fallback to defaults.

NCCL and Multi-GPU Communication: Understanding of the NCCL environment variables (NCCL_P2P_LEVEL=PHB, NCCL_IB_DISABLE=1, etc.) that configure peer-to-peer communication in a virtualized environment without NVLink.

Output Knowledge Created

This message produces several forms of output knowledge:

A testable hypothesis: That the flashinfer autotuner, when forced to profile during warmup, can discover better tile configurations for the RTX PRO 6000's SM120 architecture than the default fallback configs. This hypothesis will be tested when the server starts and the debug logs are examined.

An instrumented server configuration: The combination of --log-level debug with the existing working flags creates a diagnostic configuration that, when the server starts, will reveal the autotuner's behavior. The resulting log file (~/sglang-glm5.log) will contain the evidence needed to confirm or refute the assumptions.

A documented debugging methodology: The message demonstrates a systematic approach to performance debugging: identify the bottleneck (low GPU power utilization despite 100% utilization), trace it to kernel configuration (missing SM120 tuning config), formulate a hypothesis (autotuner can find better configs), and design an experiment (restart with debug logging to capture autotuner output).

The Thinking Process Visible in the Reasoning

The message reveals a sophisticated chain of reasoning. The assistant begins by stating the action: "Let me restart with the working config and also set FLASHINFER_AUTOTUNER_LOAD_FROM_FILE=1 to force the autotune to try harder." This immediately signals that the assistant understands the problem is one of suboptimal autotuning, not a fundamental incompatibility.

The next sentence—"But first—the autotuner only profiles when autotuning is enabled AND during warmup"—shows the assistant reasoning about the autotuner's activation conditions. This is crucial: the assistant is checking whether the conditions for autotuning are actually met in the current configuration. The discovery that disable_flashinfer_autotune=False (i.e., autotuning is NOT disabled) is reassuring—the flag is in the right state.

The final sentence—"Let me restart and capture the autotuning output"—ties the reasoning together. The assistant has identified that:

1. The autotuner should be running (flag is not disabled)
2. But we don't know what it's doing (no debug output captured)
3. Adding debug logging will reveal its behavior This is textbook diagnostic reasoning: verify the mechanism is active, then instrument to observe it. What's particularly notable is what the assistant does NOT do. It doesn't try to manually construct a tuning config file (which would require deep knowledge of the CUTLASS tile configuration space). It doesn't switch to a different MoE backend. It doesn't adjust model parallelism or try expert parallelism again. The focus is narrow and surgical: enable the existing autotuning infrastructure and observe its behavior.

Significance in the Broader Context

This message sits at a transition point in the session. The earlier phases were about getting the model to run at all—resolving NaN crashes, finding compatible attention backends, and establishing baseline throughput. Message 292 marks the beginning of the performance optimization phase, where the goal shifts from "does it work?" to "does it work efficiently?"

The outcome of this experiment will determine the next steps. If the autotuner finds better configurations and throughput improves, the path forward is clear: continue tuning, perhaps by running longer autotuning sessions or by manually constructing config files. If the autotuner produces no improvement, the assistant will need to revisit more fundamental questions about the MoE kernel selection, the communication strategy, or even the virtualization overhead identified in earlier investigations.

In a broader sense, this message illustrates a recurring challenge in deploying AI models on new hardware: the software ecosystem (flashinfer, sglang, CUTLASS) may support the architecture, but the pre-tuned configurations that make performance predictable are often missing for non-datacenter SKUs. The RTX PRO 6000, despite being a Blackwell GPU, lacks the tuning attention that B200 and GB200 receive. The assistant's workaround—forcing runtime autotuning with debug instrumentation—is a practical response to this gap, but it highlights the gap itself as a systemic issue in the ML infrastructure ecosystem.