The Autotune Awakens: Enabling FlashInfer CUTLASS MoE Kernel Tuning on SM120 Blackwell

Introduction

In the high-stakes world of large-scale ML inference deployment, few moments are as satisfying as watching a long-blocked optimization finally spring to life. Message 691 in this opencode session captures precisely such a moment: the assistant, after hours of debugging, patching, and restarting, sees the FlashInfer autotune running across all eight tensor-parallelism (TP) ranks for the GLM-5-NVFP4 model on 8x RTX PRO 6000 Blackwell GPUs. This single message—part celebration, part diagnostic check—sits at a critical inflection point in the session's arc, where the assistant transitions from enabling an optimization to verifying its execution and measuring its impact.

The Message Itself

The message reads:

Excellent! The FlashInfer autotune is running on all 8 TP ranks! This is exactly what we wanted. Let me wait for it to complete — the autotune will test various GEMM tactics for the CUTLASS MoE kernels.

>

[bash] ssh root@[REDACTED] "sleep 300 && tail -30 /root/sglang-server.log && echo '---' && wc -l /root/sglang-server.log"

>

[2026-02-19 05:56:28 TP6] HTTP Request: GET https://huggingface.co/api/models/lukealonso/GLM-5-NVFP4/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" [2026-02-19 05:56:28 TP1] HTTP Request: GET https://huggingface.co/api/models/lukealonso/GLM-5-NVFP4/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" [2026-02-19 05:56:28 TP5] HTTP Request: GET https://huggingface.co/api/models/lukealonso/GLM-5-NVFP4/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK" [2026-02-19 05:56:28 TP4] HTT...

The assistant's tone is visibly excited—"Excellent!" and "This is exactly what we wanted"—reflecting the relief and validation that comes after a complex patch finally works. The message combines an observation of success with a follow-up action: waiting 300 seconds to see the autotune complete and inspect the resulting log output.

Why This Message Was Written: The Long Road to Autotune

To understand why this message matters, we must trace the path that led here. The session had been wrestling with a persistent problem: the GLM-5-NVFP4 model, a massive mixture-of-experts (MoE) architecture with hundreds of billions of parameters, was underperforming on the 8x RTX PRO 6000 Blackwell GPUs. Throughput hovered around 880 tok/s, far below the hardware's potential.

The breakthrough insight came from analyzing a previous successful deployment of the Kimi K2-Thinking model ([msg 670]). That deployment had achieved 5,816 tok/s using a different configuration: disable_cuda_graph=True, no --max-running-requests cap, and critically, the FlashInfer CUTLASS MoE autotune enabled. The assistant realized that their current GLM-5 deployment was using --max-running-requests 64 (severely capping concurrency), had CUDA graphs enabled (which can interfere with dynamic batching), and—most importantly—was not running the FlashInfer autotune for the CUTLASS MoE backend.

The autotune was disabled because of a TODO comment in SGLang's model_runner.py (line 1837): # TODO: flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed. This comment had been left by the original developers, presumably because earlier versions of FlashInfer had compilation issues with the CUTLASS backend on certain GPU architectures. The assistant made a calculated decision to uncomment the line, enabling the autotune despite the warning ([msg 677]).

After patching the file, the assistant killed the old server and launched a new one with optimized parameters: --moe-runner-backend flashinfer_cutlass, --disable-cuda-graph, and without the --max-running-requests cap ([msg 682]). The model began loading—a process that took several minutes given the 405GB checkpoint spread across 83 safetensor shards. Message 690 showed the first sign of success: "Running FlashInfer autotune..." appeared in the logs for TP2 and TP0. Message 691 is the immediate follow-up, where the assistant confirms the autotune is running on all TP ranks and waits for it to finish.

How Decisions Were Made: The Patch and the Wait

The decision-making in this message is subtle but important. The assistant had two options after seeing the autotune start: immediately check the results, or wait a reasonable time for it to complete. The assistant chose to wait 300 seconds (five minutes), which reflects an understanding of the autotune's computational cost. FlashInfer's autotuner tests multiple GEMM (General Matrix Multiply) tactics—different combinations of block sizes, warp counts, and pipeline stages—to find the optimal kernel configuration for the specific GPU architecture. On SM120 (the compute capability of RTX PRO 6000 Blackwell GPUs), this search space is substantial, and each trial requires running actual GPU kernels.

The five-minute wait was a reasonable heuristic, but it turned out to be insufficient. The log output after 300 seconds showed the TP ranks still making HTTP requests to HuggingFace's API (GET https://huggingface.co/api/models/lukealonso/GLM-5-NVFP4/tree/main), suggesting the server was still in its initialization phase—perhaps downloading tokenizer files or configuration metadata. The autotune may not have completed yet, or if it had, the server was still processing post-autotune steps.

This reveals an important assumption: the assistant assumed the model loading and autotune would be the dominant time costs, but the server was also performing HuggingFace API calls that added unexpected latency. In a production environment behind a firewall or with cached model files, these API calls would be instantaneous, but in this LXC container setup, they introduced additional wait time.

Assumptions and Potential Mistakes

Several assumptions underpin this message, some of which proved partially incorrect:

1. The autotune would complete within 300 seconds. This was optimistic. FlashInfer's autotuner can take 10-20 minutes on large MoE models with 8 TP ranks, especially when testing CUTLASS kernels on a new architecture like SM120 where no pre-computed cache exists.
2. The autotune would improve throughput. While the K2-Thinking benchmark suggested this, the GLM-5 model uses a different architecture (DSA-based NSA attention, FP4 quantization) that may respond differently to CUTLASS MoE tuning. The autotune finds the fastest kernel for the given hardware, but it cannot fix architectural bottlenecks like PCIe P2P latency or unsupported allreduce fusion.
3. The patch was safe. The assistant acknowledged the TODO comment warning about compilation errors but proceeded anyway. This was a calculated risk—if the autotune crashed, the server would fail to start, and the assistant would revert the change. The risk paid off (the autotune ran), but the quality of the autotune results remained unknown.
4. The server was fully initialized. The log output showed "Running FlashInfer autotune..." for TP0 and TP2, but the assistant assumed all 8 ranks were running it simultaneously. The output after 300 seconds showed HTTP requests, not autotune progress, suggesting the server state was more complex than expected. A subtle mistake in the message is the phrasing "the autotune will test various GEMM tactics for the CUTLASS MoE kernels." While technically accurate, this understates the scope: the autotune also tests different quantization paths, shared-expert fusion strategies, and memory layouts. The CUTLASS MoE backend in FlashInfer is a sophisticated system that composes multiple kernel dimensions.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains:

• SGLang architecture: Understanding that model_runner.py controls the model execution pipeline, including the autotune lifecycle. The _should_run_flashinfer_autotune() method gates whether the autotune runs based on the moe_runner_backend setting.
• FlashInfer autotune system: FlashInfer uses a JIT compilation and autotuning framework that benchmarks kernel variants at startup and caches the optimal configuration. The flashinfer_cutlass backend specifically uses NVIDIA's CUTLASS library for MoE GEMM operations.
• Tensor parallelism (TP): The model is split across 8 GPUs, each running a separate process (TP0 through TP7). The autotune must run independently on each rank because each GPU may have slightly different memory latency characteristics, and the optimal kernel configuration can vary per device.
• SM120 architecture: The RTX PRO 6000 Blackwell GPUs use compute capability SM120, which differs from the datacenter Blackwell (SM100) that most FlashInfer development targets. This architecture has different shared memory sizes, warp scheduling, and instruction latencies that affect kernel tuning.
• MoE model structure: The GLM-5-NVFP4 uses mixture-of-experts layers where each token is routed to a subset of expert networks. The CUTLASS MoE backend handles the sparse matrix multiplications required for expert computation.
• The TODO comment's history: The comment # TODO: flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed. indicates a known issue that was deferred. Understanding why this comment existed—and why the assistant chose to override it—is crucial to appreciating the risk.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

1. Confirmation that flashinfer_cutlass autotune works on SM120. The TODO comment suggested it might fail, but the assistant proved it runs successfully on RTX PRO 6000 Blackwell GPUs. This is valuable for anyone deploying MoE models on consumer Blackwell hardware.
2. Evidence of server initialization latency. The HuggingFace API calls visible in the log output reveal that model serving initialization involves more than just weight loading and autotuning. Future deployments can account for this by pre-caching model metadata or using local file access.
3. A validated patch procedure. The assistant demonstrated a specific sed command to uncomment the autotune line, which can be replicated or automated. This is a concrete, reproducible optimization step.
4. Baseline timing data. The 300-second wait provides a lower bound on the autotune duration—it takes at least this long, and likely longer. Future iterations can use this to set appropriate timeout values.
5. A decision point for next steps. The message sets up the next action: once the autotune completes, the assistant will benchmark throughput at higher concurrency levels (256, 512, 1024) to measure the improvement. The current ~880 tok/s baseline will be compared against the post-autotune results.

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message reveals a structured, hypothesis-driven approach to performance optimization:

1. Observe: "The FlashInfer autotune is running on all 8 TP ranks!" — direct observation from the log output.
2. Interpret: "This is exactly what we wanted." — connecting the observation to the earlier goal (enabling the autotune).
3. Predict: "the autotune will test various GEMM tactics for the CUTLASS MoE kernels" — explaining what the autotune does, demonstrating understanding of the underlying mechanism.
4. Act: "Let me wait for it to complete" — a decision to defer further action until the autotune finishes, avoiding premature optimization.
5. Measure: The bash command with sleep 300 and tail -30 shows a deliberate measurement strategy: wait a fixed interval, then sample the log to check progress. The thinking is notably patient. Rather than polling aggressively or interrupting the autotune, the assistant allows it to run to completion. This reflects an understanding that GPU kernel autotuning is a batch operation that benefits from uninterrupted execution. The choice of tail -30 (showing the last 30 lines) rather than tail -f or continuous monitoring also reveals a design philosophy: the assistant works in discrete rounds, issuing tool calls and waiting for results. It cannot react to streaming output within a round. The 300-second sleep is a workaround for this synchronous execution model—it's a way to batch the wait into a single round.

Conclusion

Message 691 captures a moment of technical triumph tempered by the realities of distributed systems debugging. The FlashInfer autotune is running, the patch worked, and the path to higher throughput is open. But the HuggingFace API calls visible in the log output hint at the complexity still lurking: server initialization involves many moving parts, and the autotune's completion is just one milestone on a longer journey.

For the reader, this message illustrates the iterative nature of ML inference optimization: identify a bottleneck (disabled autotune), form a hypothesis (enabling it will improve throughput), implement a fix (patch the code), verify the fix (check the logs), and measure the impact (benchmark after autotune completes). Each cycle builds on the previous one, and moments like this—where a risky patch pays off—are the fuel that drives the process forward.

The assistant's willingness to override a developer TODO comment, combined with its systematic approach to verification, demonstrates the kind of engineering judgment that separates effective deployment work from blind configuration-tweaking. Not every patch works, and not every autotune run produces a faster kernel, but the discipline of observing, interpreting, and acting on evidence is what ultimately moves the needle from 880 tok/s toward the hardware's true potential.