Strategic Pivot: Transferring MoE Kernel Configs Between Blackwell Generations

Introduction

In a complex multi-GPU inference deployment session, a single message can mark the inflection point between hours of wasted computation and a targeted, efficient optimization strategy. Message 6483 in this opencode conversation is precisely such a moment. After spending significant effort attempting to run a full brute-force autotuning of Mixture-of-Experts (MoE) kernels for the Qwen3.5-122B-A10B model on NVIDIA RTX PRO 6000 Blackwell GPUs — and hitting a timeout after ten minutes on just one batch size — the assistant pivots decisively to a transfer-learning approach, leveraging existing configuration knowledge from the closely related NVIDIA B200 architecture.

The message itself is deceptively brief:

[assistant] Good B200 config. Let me take a two-pronged approach: 1. First, copy the B200 config as-is for a quick test (B200 and RTX PRO 6000 are both Blackwell, just SM100 vs SM120) 2. Then run a focused tuning with reduced search space around the B200-optimal parameters

>

Let me first copy the B200 config and see if it helps: [bash] ssh root@10.1.230.174 'mkdir -p /root/sglang-main/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_6_0 && cp "/root/sglang-main/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_4_0/E=256,N=256,device_name=NVIDIA_B200.json" "/root/sglang-main/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_6_0/E=256,N=256,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.json"' 2>&1

Beneath this straightforward action lies a rich tapestry of reasoning about GPU architecture, kernel optimization, search space reduction, and the pragmatic art of balancing theoretical optimality with practical time constraints.

The Context: A Failed Brute-Force Attempt

To understand why this message was written, one must trace back through the preceding messages in the conversation. The assistant had been working on deploying and optimizing the Qwen3.5-122B-A10B model — a 122-billion-parameter Mixture-of-Experts model with 10 billion active parameters — across four RTX PRO 6000 Blackwell GPUs. The deployment was already functional, but the assistant was pursuing performance optimization through MoE kernel autotuning.

The SGLang inference framework includes a Triton-based MoE kernel implementation that can be tuned for specific GPU architectures. The tuning process involves testing hundreds of configurations — varying block sizes, warp counts, and pipeline stages — across multiple batch sizes to find the optimal kernel parameters for a given hardware target. The tuning script (tuning_fused_moe_triton.py) tests 1,920 configurations per batch size, with 100 timing iterations each.

In [msg 6479], the assistant launched this full autotuning process with a single batch size on one GPU. The command timed out after exceeding 600 seconds (10 minutes) without producing any output. This was the critical failure that forced a strategic rethink. The assistant correctly diagnosed the problem: "The tuning is too slow with 1920 configs x 18 batch sizes on a single GPU." With 18 batch sizes to test, a naive full search would take over three hours per GPU — and that's before considering that the script uses Ray to parallelize across GPUs, adding orchestration overhead.

The Strategic Pivot: Two-Pronged Approach

The subject message represents the assistant's response to this failure. Rather than simply retrying with more GPUs or waiting longer, the assistant articulates a two-pronged strategy that demonstrates deep understanding of both GPU architecture and practical optimization workflows.

Prong 1: Copy the B200 config as-is for immediate testing. This is the pragmatic gambit. The NVIDIA B200 and the RTX PRO 6000 Blackwell are both based on the Blackwell GPU architecture — the B200 uses the SM100 compute capability, while the RTX PRO 6000 uses SM120. The assistant explicitly acknowledges this difference ("just SM100 vs SM120") but correctly reasons that the architectural similarities are likely sufficient for the configuration to be functional, if not perfectly optimal. This is a classic engineering trade-off: deploy a "good enough" solution immediately rather than wait for a perfect one.

Prong 2: Run a focused tuning with a reduced search space around the B200-optimal parameters. This is the refinement step. Instead of testing all 1,920 configurations from scratch, the assistant plans to use the B200's optimal parameters as a center point and search locally around them. This dramatically reduces the search space — from thousands of configurations to perhaps a few dozen — while still allowing the tuning process to discover SM120-specific optimizations.

The two prongs are complementary rather than sequential. The first prong provides immediate value (the server can be restarted with a potentially better configuration right away), while the second prong promises a more refined solution later. This dual-track approach is characteristic of experienced systems engineers: always have a working fallback while pursuing improvement.## The Reasoning: Why Transfer Learning Works for GPU Kernel Configs

The assistant's decision to borrow the B200 configuration is not arbitrary — it is grounded in a sophisticated understanding of GPU microarchitecture and the Triton compiler's behavior. Triton kernels are expressed in a tile-based programming model where the programmer specifies block sizes (M, N, K dimensions), the number of warps per block, and the number of pipeline stages. These parameters interact with the GPU's shared memory size, register file capacity, memory bandwidth, and compute throughput in complex ways.

The Blackwell architecture introduced several key innovations over Hopper (H100): fourth-generation Tensor Cores with FP8/FP4 support, a 50% larger shared memory (228 KB per SM versus 192 KB on Hopper), and enhanced data compression capabilities. However, the fundamental SM organization — the number of warps per SM, the memory hierarchy, and the instruction pipeline — remained largely consistent across Blackwell variants. The B200 (SM100) and RTX PRO 6000 (SM120) share the same Blackwell DNA; the SM version difference primarily reflects feature set variations (such as FP4 support) rather than a fundamentally different execution model.

By copying the B200 config, the assistant is making a reasoned assumption: the optimal block sizes and warp configurations for a Blackwell GPU with E=256 experts and N=256 intermediate size will be similar enough across SM100 and SM120 that the B200 config will either work well or provide an excellent starting point for refinement. The config file itself reveals the B200-optimal parameters: BLOCK_SIZE_M=16, BLOCK_SIZE_N=64, BLOCK_SIZE_K=64, GROUP_SIZE_M=1, num_warps=4, num_stages=5 for batch size 1. These values encode specific trade-offs — the relatively small M dimension (16) suggests the kernel is optimized for the low-occupancy regime of single-request inference, while the 4:1 ratio of N to M reflects the wide intermediate dimension of the MoE layer.

The Assumptions Embedded in the Message

Every engineering decision rests on assumptions, and this message contains several worth examining:

Assumption 1: SM100 and SM120 configurations are interchangeable. This is the central assumption of the entire approach. While architecturally similar, SM120 on the RTX PRO 6000 includes support for FP4 quantization that SM100 on the B200 lacks. The presence of FP4 Tensor Core instructions could affect instruction scheduling, register pressure, and memory access patterns in ways that might shift the optimal kernel configuration. The assistant implicitly assumes these differences are second-order effects that won't invalidate the B200 config.

Assumption 2: The config directory triton_3_6_0 is the correct target. The source config lives under triton_3_4_0 (referring to Triton compiler version 3.4.0), while the destination is triton_3_6_0. The assistant is assuming that the Triton 3.6.0 compiler can consume configs generated for Triton 3.4.0 without issues. This is likely true — the config format is a simple JSON dictionary mapping batch sizes to parameter tuples — but it's an assumption worth noting.

Assumption 3: The file naming convention will be recognized by the config loader. The config file is named E=256,N=256,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.json. The assistant is assuming that SGLang's config loader will match this device name against the GPU's reported name. If the device name string doesn't match exactly (e.g., if the driver reports a slightly different name), the config will be silently ignored and the fallback default will be used instead.

Assumption 4: The B200 config was itself well-tuned. The assistant has no way to verify that the B200 config file represents genuinely optimal parameters rather than a suboptimal or default configuration. It's trusting that the SGLang development team or community performed proper tuning for the B200.

Input Knowledge Required to Understand This Message

A reader needs substantial background knowledge to fully grasp the significance of this message:

  1. Mixture-of-Experts architecture: Understanding that models like Qwen3.5-122B-A10B use sparse MoE layers where each token is routed to a subset of experts (8 out of 256 in this case), and that the expert computation is the dominant cost.
  2. Triton compiler and kernel tuning: Knowledge that Triton is a GPU kernel programming language that compiles tile-based computations, and that its performance depends critically on block size parameters that must be tuned per architecture.
  3. NVIDIA GPU architecture lineage: Understanding the Blackwell architecture family, the distinction between SM100 (B200, datacenter) and SM120 (RTX PRO 6000, workstation), and what architectural features they share versus what differs.
  4. SGLang's MoE kernel infrastructure: Familiarity with the config file system under fused_moe_triton/configs/, the naming convention (E=256,N=256,device_name=...), and the fallback behavior when no matching config exists.
  5. The model's dimensions: Knowing that Qwen3.5-122B-A10B has 256 experts with an intermediate size of 1024 (sharded to 256 per GPU with TP=4) and hidden size 3072, which determines the E=256,N=256 config file name.

Output Knowledge Created

This message produces several concrete artifacts and insights:

  1. A new config file at triton_3_6_0/E=256,N=256,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.json containing the B200-optimal parameters. This file is immediately usable by the SGLang runtime when it loads the MoE kernel for the RTX PRO 6000.
  2. A documented strategy (the two-pronged approach) that can be applied to other GPU architectures and model configurations. The pattern — "borrow from nearest relative, then refine locally" — is a generalizable optimization methodology.
  3. A time budget saved: By avoiding the full 1920-config × 18-batch-size search, the assistant saves hours of GPU compute time. Even if the B200 config is not perfectly optimal, the opportunity cost of running the full search is eliminated.
  4. A baseline for comparison: Once the server is restarted with the B200-derived config, the assistant can benchmark throughput and compare against the previous default configuration. Any improvement validates the approach; any regression signals that SM120-specific tuning is essential.

The Thinking Process: A Case Study in Adaptive Problem-Solving

The reasoning visible in the messages leading up to and including this one reveals a sophisticated problem-solving approach. The assistant begins with the "textbook" solution — run the full autotuning script as designed. When this fails (timeout), it doesn't simply retry with different parameters; it performs a root cause analysis: "1920 configs × 18 batch sizes on a single GPU" is computationally prohibitive.

The pivot to B200 configs is not the first alternative considered. In [msg 6475], the assistant considered using the separate tuning script (tuning_fused_moe_triton_sep.py) which handles both up and down MoE kernels separately, but discovered it required a --topk-ids-dir argument with real expert routing data — adding another dependency. In <msg id=6477-6478>, the assistant patched common_utils.py to fix a bug where the architecture was lost after the text_config redirect, demonstrating willingness to modify the toolchain rather than accept its limitations.

Each failure narrows the search space. The timeout on the full tuning → consider B200 transfer. The missing --topk-ids-dir → use the simpler unified script. The architecture bug → patch the script. This is adaptive problem-solving in action: each obstacle is met with a targeted response that preserves forward progress.

The two-pronged approach itself embodies a key insight about optimization work: perfect is the enemy of good enough. The assistant could have spent hours debugging the tuning script, fixing the Ray networking, and running the full search. Instead, it chose to deploy a "probably good" configuration immediately while preserving the option to refine later. This is the mark of an engineer who understands that in production systems, a 90% solution today is worth more than a 100% solution next week.

Conclusion

Message 6483 captures a pivotal moment in a complex GPU optimization session — the transition from a failed brute-force approach to a targeted, knowledge-driven strategy. By recognizing the architectural kinship between B200 and RTX PRO 6000 Blackwell GPUs, the assistant makes a calculated bet that pays off in saved time and immediate utility. The message is a testament to the power of architectural reasoning in systems optimization: understanding not just what a GPU is, but what it shares with its relatives, enables intelligent shortcuts that pure computation cannot match.

The two-pronged approach — deploy the known-good config immediately, then refine locally — is a pattern that extends far beyond GPU kernel tuning. It appears in everything from compiler optimization (use -O2 first, then profile-guided optimization) to machine learning hyperparameter tuning (start with known-good architectures, then Bayesian optimization). In this sense, the message is not just about copying a JSON file; it's about a philosophy of engineering that values time, leverages knowledge, and always keeps a working system within reach.