The Pragmatic Autotuner: Borrowing B200 MoE Kernel Configs for RTX PRO 6000 Blackwell
In the middle of an intense optimization session for a Qwen3.5-122B-A10B-FP8 model running on four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, the assistant reaches a quiet but significant milestone. Message [msg 6491] is deceptively brief — just a few lines of commentary followed by a benchmark run — but it represents the payoff of a long chain of troubleshooting, patching, and pragmatic decision-making. The assistant has just deployed MoE (Mixture-of-Experts) kernel configurations borrowed from an NVIDIA B200 GPU onto the RTX PRO 6000 hardware, and the benchmark results confirm that the gamble paid off.
The Road to This Message
To understand why this message was written, we must trace the assistant's trajectory through the preceding messages. The session had been focused on squeezing maximum throughput from the Qwen3.5-122B-A10B-FP8 model, a 122-billion-parameter MoE model with 256 experts and an intermediate size of 1024. Earlier optimization attempts had been frustrating: flags like --enable-fused-moe-sum-all-reduce and --enable-flashinfer-allreduce-fusion produced no measurable improvement (see [msg 6463]). The assistant correctly diagnosed that the flashinfer allreduce fusion wasn't even activating on the SM120 architecture. With the low-hanging fruit exhausted, the assistant turned to the one optimization that promised real gains: MoE Triton kernel autotuning.
The autotuning effort quickly ran into obstacles. The tuning script (tuning_fused_moe_triton.py) crashed because it failed to extract the model architecture from the Qwen3.5 config — the text_config redirect left config.architectures as None ([msg 6475]). The assistant patched common_utils.py to capture the architecture before the redirect ([msg 6478]). But even after fixing the bug, the tuning proved impractical: with 1920 configurations to test across 18 batch sizes, a single run timed out after 10 minutes ([msg 6479]). Running the full autotuning across all four GPUs would have consumed hours of compute time.
The Pragmatic Pivot
This is where the assistant's engineering judgment comes into sharp focus. Rather than brute-forcing the autotuning, the assistant recognized an opportunity for a shortcut. Both the RTX PRO 6000 Blackwell Server Edition and the NVIDIA B200 are built on the Blackwell architecture — SM120 versus SM100, but with similar microarchitectural characteristics. The B200 already had a tuned MoE kernel config for the exact dimensions needed (E=256, N=256). The assistant's reasoning, articulated in [msg 6482], was clear:
"Let me take a different approach — instead of the full brute-force autotuning, let me use the B200 configs as a starting point (same Blackwell arch, SM100 vs SM120, but similar microarchitecture) and adapt them."
This is a textbook example of transfer learning applied to systems optimization: leveraging configurations tuned for a similar hardware target rather than starting from scratch. The assistant copied the B200 config file into the correct path for the RTX PRO 6000 device name ([msg 6483]), verified the device name matched SGLang's expectations ([msg 6484]–[msg 6486]), and restarted the server.
What the Subject Message Actually Says
The subject message itself is the verification step. The assistant checks the server logs first:
"No warnings at all — both configs were found (it falls back to the same config for down_moe)."
This is an important detail. The MoE kernel config system in SGLang has two variants: the "up/gate" kernel and the "down" kernel. The assistant had previously noted that when no separate _down.json config exists, the system falls back to the regular config ([msg 6475]). The absence of warnings confirms that the fallback mechanism worked correctly — the B200-derived config is being used for both kernel variants.
Then comes the benchmark. The assistant runs bench_qwen.py with concurrency levels of 1, 4, 16, 32, and 64:
C | Agg tok/s | Per-req tok/s | Tokens | Wall(s) | Ok
-----------------------------------------------------------------
1 | 122.9 | 124.3 | 4000 | 32.6 | 4
4 | 410.6 | 118.9 | 7197 | 17.5 | 8
16 | 1210.5 | 84.9 | 29501 | 24.4 | 32
32 | 1514.7 | 62.5 | 51847 | 34.2 | 64
64 | 1566.1 | 65.3 | 97000 | 61.9 | 128The numbers tell a clear story. Single-request throughput is 122.9 tok/s aggregate (124.3 per-request). As concurrency increases, aggregate throughput scales nearly linearly up to C=16 (1210.5 tok/s), then begins to saturate. At C=64, the system delivers 1566.1 tok/s aggregate — roughly 12.7× the single-request throughput. The per-request throughput drops from 124.3 tok/s at C=1 to 65.3 tok/s at C=64, indicating the GPUs are well-utilized but individual requests experience longer queueing and scheduling delays.
Assumptions and Their Validity
The assistant made several assumptions in this message and the preceding work. The most significant was that the B200 kernel config would perform well on the RTX PRO 6000 Blackwell. This assumption rested on architectural similarity: both are Blackwell-family GPUs, sharing the same Tensor Core generation and similar memory hierarchies. The SM100 (B200) and SM120 (RTX PRO 6000) differ in compute capability version and some microarchitectural details, but the Triton kernel parameters — block sizes, warp counts, pipeline stages — are coarse enough that they transfer reasonably well across Blackwell variants. The benchmark results validated this assumption.
A second assumption was that the down_moe config fallback would be harmless. The assistant reasoned that since the up and down kernels have the same dimensions (E=256, N=256), the same block tiling parameters would work for both. The absence of performance warnings or errors in the logs confirmed this.
A third, more subtle assumption was that the benchmark itself was representative. The assistant used a fixed prompt and generation length (implied by the 100-token warmup and the ~4000 total tokens at C=1). Real-world workloads with variable-length inputs and outputs might see different scaling behavior. The benchmark also doesn't show tail latency or time-to-first-token metrics, which matter for interactive applications.
Input Knowledge Required
To fully understand this message, one needs knowledge of several domains. First, the Mixture-of-Experts architecture: the Qwen3.5-122B-A10B model has 256 experts with an intermediate size of 1024 and a hidden size of 3072, routing 8 experts per token. The MoE kernel's performance is highly sensitive to tiling parameters (BLOCK_SIZE_M/N/K, GROUP_SIZE_M, num_warps, num_stages) that must be tuned for the specific hardware and model dimensions.
Second, SGLang's configuration system: the framework stores per-device MoE kernel configs as JSON files in a directory tree organized by Triton version and device name. The get_config_file_name() function constructs filenames like E=256,N=256,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.json. The down_moe variant appends _down to the filename.
Third, the Blackwell GPU architecture: understanding that B200 (SM100) and RTX PRO 6000 (SM120) share the same fundamental Blackwell design but differ in compute capability version, memory configuration, and target market (datacenter vs professional workstation).
Output Knowledge Created
This message produces several valuable outputs. Most concretely, it provides a throughput benchmark for the Qwen3.5-122B-A10B-FP8 model on four RTX PRO 6000 Blackwell GPUs with the B200-derived MoE kernel config. The benchmark establishes a baseline: ~122 tok/s single-request, scaling to ~1566 tok/s at C=64 concurrency. These numbers serve as a reference point for future optimization work.
The message also implicitly validates the approach of cross-device config transfer within the same GPU architecture family. This is a reusable insight: when full autotuning is impractical, borrowing configs from a similar device can yield good results with minimal effort. The assistant's decision to check for warnings before benchmarking is also instructive — it demonstrates a disciplined verification workflow that catches configuration errors before they contaminate benchmark results.
The Thinking Process
The assistant's reasoning is visible in the structure of the message itself. The first sentence — "No warnings at all — both configs were found" — shows the assistant checking the precondition before proceeding to the benchmark. This is a deliberate ordering: there's no point benchmarking if the config wasn't loaded correctly. The parenthetical clarification "(it falls back to the same config for down_moe)" shows the assistant reasoning about the config system's behavior and confirming that the fallback path is acceptable.
The benchmark invocation itself is carefully designed. The assistant chooses concurrency levels that span the expected range: 1 (single request), 4 (matching the 4-GPU topology), 16, 32, and 64 (high concurrency). This produces a scaling curve that reveals both the peak throughput and the saturation point. The warmup step (100 tokens) ensures the GPU caches and kernel launch overheads are stabilized before measurement.
The message ends with a trailing ellipsis and no further commentary — the benchmark results speak for themselves. The assistant doesn't need to interpret them because the numbers are unambiguous: the config transfer worked, and the system is performing well. This understated delivery is characteristic of a practitioner who trusts their data and lets the numbers tell the story.
Conclusion
Message [msg 6491] captures a moment of quiet validation in a complex optimization journey. The assistant navigated a buggy tuning script, an impractical autotuning workload, and an unfamiliar config system to arrive at a pragmatic solution: borrowing MoE kernel configurations from a sibling Blackwell GPU. The benchmark confirmed that the shortcut was sound, delivering 122–1566 tok/s across concurrency levels. This message exemplifies the kind of engineering judgment that separates effective system optimization from mere trial-and-error — knowing when to brute-force, when to pivot, and when a well-reasoned shortcut is the best path forward.