The Turning Point: MoE Kernel Autotuning on Blackwell GPUs

In the middle of a long optimization session for the Qwen3.5-122B-A10B model running on SGLang across four NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant reaches a critical inflection point. Message 6472 is short — barely two lines of reasoning followed by a single bash command — but it represents the culmination of a lengthy investigative arc and the decisive pivot toward the session's most impactful optimization.

The Message

[assistant] So we need two config files: 1. E=256,N=256,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.json — up/gate MoE 2. E=256,N=256,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition_down.json — down MoE

>

The separate tuning script tunes both. Let me use that one. First, let me stop the server and kill zombie processes. [bash] ssh root@10.1.230.174 'systemctl stop sglang-qwen.service' 2>&1

The Context: A Trail of Dead Ends

The assistant had been chasing quick wins. Over the preceding messages, it tested three optimization flags in rapid succession. First came --enable-fused-moe-sum-all-reduce, which showed no measurable improvement in throughput benchmarks. Then --enable-flashinfer-allreduce-fusion was tried, but a deep-dive task into the SGLang source code revealed that the fusion logic explicitly checks for compute capability SM90 (H100) or SM100 (Hopper Next) — SM120 (Blackwell) was not included. The flag was a no-op on these GPUs, yet it had a pernicious side effect: it forced --disable-piecewise-cuda-graph and reduced max_running_requests from 48 to 26, actively harming throughput. The assistant removed both flags and restored the original configuration, but the max_running_requests stayed at 26 — a mystery that was never fully resolved, possibly an inherent effect of speculative decoding in this SGLang version.

These experiments consumed roughly a dozen messages and produced a clear negative result: the low-hanging fruit had already been picked. Any further performance gains would require deeper intervention.

The Discovery: Missing MoE Kernel Configs

The critical clue came from the server's startup logs. Sandwiched among the usual initialization messages were two warnings: "Performance might be sub-optimal!" for both the "up" and "down" fused MoE Triton kernels. SGLang's Mixture-of-Experts implementation uses Triton kernels with pre-tuned configuration files that specify optimal block sizes, warp counts, and pipeline stages for each GPU architecture. These configs live in python/sglang/srt/layers/moe/fused_moe_triton/configs/, organized by Triton version.

The assistant discovered that the server had Triton 3.6.0 installed (via the ML environment's Python), but the configs directory contained only triton_3_1_0 through triton_3_5_1 — no triton_3_6_0 directory existed. Worse, even within the existing version directories, there were no configs for the NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition device. The Blackwell architecture (SM120) was simply too new; no pre-tuned configurations had been contributed upstream.

This is a common challenge when deploying cutting-edge hardware: the software ecosystem lags behind. The RTX PRO 6000 Blackwell GPUs, with their 96GB of VRAM and SM120 compute capability, were bleeding-edge enough that the Triton kernel autotuning infrastructure had not yet catalogued them. The only path forward was to generate the configs locally.

The Reasoning: Connecting Source Code to Action

The assistant's reasoning in message 6472 is deceptively compact. It synthesizes information gathered across multiple prior investigations into a precise action plan. The config file names are not guessed — they are derived from the naming convention hardcoded in fused_moe_triton_config.py, which the assistant had read in message 6471. That source code revealed:

down_moe_selector = "_down" if down_moe else ""
return f"E={E},N={N},device_name={device_name}{dtype_selector}{block_shape_selector}{per_channel_quant_selector}{down_moe_selector}.json"

The parameters E=256,N=256 come from the Qwen3.5-122B-A10B model architecture: 256 experts with an intermediate size of 256 per expert (after the A10B pruning). The device name is the exact string reported by PyTorch for these GPUs. The _down suffix distinguishes the down-projection MoE kernel from the up/gate kernel.

The assistant also makes a critical decision about which tuning script to use. There are two scripts in the repository: tuning_fused_moe_triton.py (515 lines) and tuning_fused_moe_triton_sep.py (which the assistant had confirmed exists in message 6469). The separate script explicitly handles both down_moe=False and down_moe=True cases in a single run, as seen in the source code lines trace0 = BestConfigTrace("kernel0", down_moe=False) and trace1 = BestConfigTrace("kernel1", down_moe=True). Choosing the separate script avoids having to run the tuning process twice.

The Assumptions Underlying the Decision

Several assumptions are embedded in this message. First, the assistant assumes that the tuning script will work correctly on SM120 Blackwell GPUs. The Triton kernels and the tuning infrastructure were likely developed on earlier architectures (SM80 Ampere, SM90 Hopper), and there is no guarantee that the search space of block sizes and warp configurations will produce valid results on the new architecture. The assistant implicitly trusts that Triton's JIT compilation will handle SM120 correctly.

Second, the assistant assumes that the tuning process can complete within a reasonable time frame. The search space spans hundreds of configurations, and each configuration must be benchmarked across multiple batch sizes. On a 4-GPU system, this could take hours. The assistant does not estimate the runtime or set expectations.

Third, there is an assumption that the tuned configs will actually produce a meaningful speedup. The "Performance might be sub-optimal!" warning is generic — it fires whenever no tuned config exists, regardless of whether the default fallback is actually slow. The assistant is betting that the default Triton heuristics are significantly suboptimal for the unusual combination of Blackwell's SM120 architecture and the Qwen3.5 MoE dimensions.

The Action: Taking the Server Offline

The bash command in this message — systemctl stop sglang-qwen.service — is itself a significant decision. The assistant had been running a live production service. Stopping it means accepting downtime. The message also mentions "kill zombie processes," acknowledging that a simple systemd stop might not be sufficient to fully release GPU memory. The subsequent message (msg 6473) confirms this concern: the assistant runs a more aggressive cleanup, including fuser -k /dev/nvidia* to kill any lingering processes holding GPU resources.

This downtime is the cost of doing the MoE kernel autotuning, which requires exclusive GPU access. The assistant could have attempted to run the tuning on a single GPU while leaving the server running on the other three, but the tuning script uses Ray to distribute work across all available GPUs. Running it on a subset would require code modification. The assistant chooses the simpler, more reliable path: stop everything, tune, then restart.

The Knowledge Flow: Inputs and Outputs

The input knowledge required to understand this message is substantial. One must know what MoE (Mixture-of-Experts) kernels are and why they need architecture-specific tuning. One must understand the SGLang codebase structure — that config files live in a versioned directory and are loaded by filename pattern. One must know the Qwen3.5-122B-A10B model architecture: that it has 256 experts and uses a pruned intermediate size. One must understand the GPU naming convention (NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition) and how it maps to the device_name field in PyTorch. And one must understand the difference between "up/gate" and "down" projections in an MoE block — the gate projection routes tokens to experts, the up projection computes the expert's hidden states, and the down projection maps back to the model dimension.

The output knowledge created by this message is the explicit identification of the two config files that need to be generated, the selection of the appropriate tuning script, and the initiation of the server shutdown sequence. This knowledge is immediately actionable — the next messages in the session will run the tuning script, wait for it to complete, copy the generated configs into the correct directory, and restart the server with optimized kernels.

The Thinking Process: A Study in Synthesis

What makes this message interesting is the compression of reasoning. The assistant does not re-explain the source code analysis, does not recount the failed flag experiments, and does not justify why autotuning is the next logical step. It simply states the conclusion: "So we need two config files" — as if the answer were obvious. But this apparent simplicity masks a complex chain of inference.

The assistant had to:

  1. Notice the "Performance might be sub-optimal!" warnings in the server logs
  2. Investigate the config directory structure and discover the missing Triton version
  3. Read the config file naming logic in the source code to understand the pattern
  4. Extract the model's MoE parameters (E=256, N=256) from the model configuration
  5. Identify the device name string that PyTorch reports for these GPUs
  6. Discover that two separate config files are needed (up and down)
  7. Find and evaluate the two tuning scripts to determine which one handles both cases
  8. Decide to stop the server and proceed with tuning Each of these steps required tool calls, source code reading, and synthesis. The message is the tip of an iceberg — the visible output of a much larger reasoning process that unfolded across the preceding dozen messages.

The Broader Significance

This message represents a pattern that recurs throughout the opencode session: the assistant systematically exhausts quick optimization attempts, then commits to a more invasive but potentially more rewarding intervention. The failed flag experiments were not wasted — they ruled out the easy paths and built confidence that the MoE kernel tuning was the right next step. The assistant's willingness to take the server offline reflects a pragmatic assessment: the tuning cost (downtime) is justified by the expected benefit (better throughput for all future inference).

Whether the MoE kernel autotuning ultimately succeeds is a question answered in subsequent messages. But message 6472 captures the moment of commitment — the pivot from investigation to action, from analysis to execution. It is a small message with large consequences.