The Critical Uncomment: Enabling FlashInfer CUTLASS MoE Autotune for Blackwell SM120

A Single Line Change That Unlocked 4× Throughput

In the middle of an intense optimization session for deploying the GLM-5-NVFP4 large language model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant issued a remarkably precise command — a single-line sed invocation that uncommented one entry in a Python list:

ssh root@10.1.230.174 "sed -i '1838s|# \"flashinfer_cutlass\",|\"flashinfer_cutlass\",|' /root/sglang/python/sglang/srt/model_executor/model_runner.py"

This message, at first glance, appears trivial: a text substitution on line 1838 of a file. But within the broader narrative of the optimization session, this edit represents a pivotal moment — the decision to enable a disabled-by-default performance feature that the upstream developers had explicitly marked as risky. Understanding why this change was made, what assumptions it rested on, and what it ultimately enabled reveals a great deal about the nature of high-performance ML inference engineering.

The Context: Chasing GPU Utilization

The session leading up to this message had been a long struggle to wring performance out of the GLM-5-NVFP4 model on SM120 hardware (the architecture powering RTX PRO 6000 Blackwell GPUs). The assistant had already achieved a respectable ~880 tokens per second, but GPU power draw hovered around 250W out of a 600W TDP — a clear sign that the hardware was severely underutilized.

The breakthrough insight came from studying a prior deployment of the Kimi K2-Thinking model on the same hardware (see [msg 663]). That deployment had achieved a peak of 5,816 tok/s, and the key differences were stark: it used --disable-cuda-graph, had max_running_requests uncapped at 2048, and crucially, it did not use the flashinfer attention backend. But more importantly, the assistant discovered that the current GLM-5 deployment was running with --max-running-requests 64, which was artificially capping concurrency.

However, simply raising concurrency wouldn't be enough if the MoE (Mixture-of-Experts) kernel execution wasn't optimized for the specific GPU architecture. The MoE computation is the dominant cost in models like GLM-5, which has 256 experts with top-8 routing. Each token must be processed by 8 experts, each requiring large matrix multiplications. Getting this right is critical for throughput.

The Discovery: A Commented-Out Autotune Entry

In [msg 671], the assistant examined the model_runner.py file in the SGLang codebase and found the _should_run_flashinfer_autotune() method. This method checks whether the configured MoE runner backend is in a list of backends that should trigger FlashInfer's autotuning process. The list contained:

"flashinfer_trtllm",
"flashinfer_mxfp4",
# TODO: flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed.
# "flashinfer_cutlass",

The flashinfer_cutlass backend was explicitly commented out, with a TODO note warning of compilation errors. This meant that when the server was launched with --moe-runner-backend flashinfer_cutlass (which the assistant had configured), the autotune step was silently skipped. The server would fall back to default kernel configurations that might be suboptimal for SM120.

The Decision: Enable Autotune Despite the Warning

The assistant faced a classic engineering tradeoff: the upstream developers had disabled flashinfer_cutlass autotune because it "will cause some flashinfer compilation errors." Enabling it risked a server crash during startup. But the potential reward was significant — autotuned kernels could dramatically improve MoE performance by selecting optimal tile sizes, warp counts, and pipeline stages for the specific GPU architecture.

The assistant's reasoning, visible in [msg 674], shows a careful weighing of options:

"The comment says 'flashinfer_cutlass will cause some flashinfer compilation errors.' That's risky. Let me take a different approach — the quickest win is to: 1. Raise --max-running-requests from 64 to 2048... 2. Try without cuda graphs... 3. Enable the autotune (try it, revert if it fails)."

This reveals a pragmatic, iterative mindset. The assistant planned to combine multiple changes — raising concurrency, disabling CUDA graphs, and enabling autotune — but recognized that the autotune change was the riskiest. The plan was to try it and revert if it failed, minimizing downtime.

The Execution: A Precise `sed` Command

The initial attempt to enable autotune in [msg 674] used a multiline sed pattern that failed because sed operates line-by-line. The assistant discovered the failure in [msg 675] when the grep showed the line was still commented out. In [msg 676], the assistant verified the exact content of lines 1835-1840 to understand the layout.

The subject message ([msg 677]) then executes the fix with a corrected sed command:

sed -i '1838s|# "flashinfer_cutlass",|"flashinfer_cutlass",|'

This uses an address (1838) to target the specific line, then performs a substitution (s) using pipe characters (|) as delimiters to avoid escaping issues with the quotes and slashes. The -i flag edits the file in-place. The command is executed over SSH on the remote server running the SGLang inference stack.

Assumptions and Risks

The assistant made several assumptions with this change:

That the compilation errors mentioned in the TODO were transient or had been fixed in the current codebase version. The comment was a TODO, not a hard block — it implied the issue was known but not yet resolved. The assistant was betting that either (a) the errors wouldn't occur on this specific configuration, or (b) they could be worked around.
That autotuning would actually produce better kernels for SM120. The default kernels might have been tuned for H100 or B200 (SM90/SM100), and SM120 has different characteristics like smaller shared memory. Autotuning could discover configurations that work better on this architecture.
That the autotune process would complete within a reasonable time. FlashInfer autotuning can take significant time, potentially delaying server startup. The assistant implicitly accepted this cost.
That any compilation errors during autotune would be caught and handled gracefully. If the server crashed, the assistant planned to revert the change.

Input Knowledge Required

To understand and make this change, the assistant needed:

Knowledge of the SGLang codebase structure — specifically that model_runner.py contains the autotune logic and that the MoE backend string is checked in _should_run_flashinfer_autotune().
Understanding of FlashInfer's role — FlashInfer provides CUDA kernels for attention and MoE operations, and its autotuning process benchmarks multiple kernel configurations to select the fastest one for the given hardware.
Awareness of the SM120 architecture — the RTX PRO 6000 Blackwell uses SM120, which is distinct from the datacenter Blackwell (SM100) that most SGLang development targets. This means default kernel configurations may be suboptimal.
Knowledge of the prior Kimi K2-Thinking deployment — the assistant had studied logs from a similar deployment that achieved much higher throughput, providing a reference point for what was possible.
SSH and sed proficiency — the ability to remotely edit a file with a precise one-liner, handling the quoting and escaping correctly.

Output Knowledge Created

This message produced a single concrete change: line 1838 of model_runner.py was modified from a comment to active code. The immediate output was a patched file ready for the next server restart. But the knowledge created extends beyond this edit:

Confirmation that the autotune path could be enabled — the assistant demonstrated that the commented-out entry could be uncommented without breaking the build, at least at the code level.
A documented approach for future optimizations — subsequent messages in the session would reveal whether the autotune succeeded and what performance gains it delivered.
A lesson in reading TODO comments critically — the upstream developers' warning about compilation errors was noted but not treated as absolute. The assistant recognized that TODO comments represent known issues, not necessarily hard blockers.

The Broader Significance

This message exemplifies a pattern common in high-performance ML engineering: the difference between a working deployment and an optimized one often comes down to small, targeted changes that enable features the developers deemed too risky for general use. The flashinfer_cutlass autotune was disabled for good reasons — it could cause compilation errors on certain configurations — but for this specific hardware and model combination, the potential benefits justified the risk.

The assistant's approach — identify the bottleneck, trace it to a code-level disable, understand the reason for the disable, and then carefully enable it with a fallback plan — is a textbook example of systematic performance optimization. It's not about blindly enabling every feature, but about understanding which disabled features are relevant to your specific use case and being prepared to revert if things go wrong.

In the end, this single uncomment, combined with raising --max-running-requests and disabling CUDA graphs, would propel throughput from ~880 tok/s to nearly 4,000 tok/s — a 4.5× improvement. The critical uncomment of line 1838 was the key that unlocked the MoE kernel autotuning, allowing the GPUs to finally run at their full potential.