The Kernel Tuning Imperative: Why a New GPU Demands Custom Configs

"Also probaby tune the kernels, this is really new gpu"

This single sentence from the user, appearing at message index 280 in a complex deployment session, is a masterclass in strategic redirection. On its surface, it is a brief suggestion — barely six words — but within the context of the conversation, it carries enormous weight. It arrives at a critical juncture where the assistant has been methodically exploring alternative MoE (Mixture-of-Experts) runner backends, swapping one kernel implementation for another, without fundamentally improving throughput. The user's message cuts through this trial-and-error approach with a deeper insight: the problem isn't which backend you choose, but whether any backend has been tuned for the hardware you're running on.

The Moment of Intervention

To understand why this message was written, we must reconstruct the state of the conversation at that precise moment. The assistant had just attempted to launch the SGLang server with --moe-runner-backend flashinfer_trtllm ([msg 278]), hoping that a different MoE kernel implementation would improve the disappointing throughput numbers. The server crashed with a traceback ([msg 279]), revealing that the trtllm_fp4_block_scale_moe path was incompatible. The assistant was about to pivot to yet another backend (flashinfer_cutedsl) and continue the cycle of swapping implementations.

The user intervenes here because they recognize a pattern: the assistant is treating the problem as a selection problem (which backend to choose) when it is actually a configuration problem (how to tune the chosen backend for the specific hardware). The RTX PRO 6000 Blackwell (SM120 architecture) is not just another GPU — it is a brand-new chip with its own optimal tile sizes, shared memory configurations, and warp scheduling characteristics. The kernel configurations that work well on datacenter Blackwell (B200, GB200 — SM100 architecture) may be suboptimal or even incorrect on the workstation variant.

The Reasoning Behind the Suggestion

The user's reasoning draws on several threads that had been developing in the conversation. Earlier, the assistant had discovered that flashinfer's tuning configs directory contained only two files: v0_1_trtllm_fused_moe_NVIDIA_B200.py and v0_1_trtllm_fused_moe_NVIDIA_GB200.py ([msg 285]). When the autotuner tried to load a config for the RTX PRO 6000 Blackwell Server Edition, the file path resolved to v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.py — which did not exist. The autotuner fell back to default config 0 with tile_config -1, meaning no hardware-specific tuning was applied.

The user connected this absence of tuning configs to the performance data the assistant had been collecting. The GPUs were showing 100% utilization but only drawing 55% of their 600W power budget ([msg 268]). This is a classic symptom of suboptimal kernel configurations: the GPU is busy but not productively occupied — it's spending cycles on inefficient memory access patterns, underutilized tensor cores, or excessive kernel launch overhead. The kernels are running, but they're not running well.

Furthermore, the user understood that the SM120 architecture (RTX PRO 6000) is architecturally distinct from SM100 (B200/GB200). While both are "Blackwell," the SM120 variant has different cache sizes, different warp scheduling capabilities, and different optimal tile configurations for CUTLASS-based GEMM operations. The flashinfer library had dedicated SM120 CUDA files (fp4_gemm_cutlass_sm120.cu, group_gemm_sm120_binding.cu, gemm_groupwise_sm120.cu) — discovered by the assistant in [msg 283] — but no corresponding tuning configs to tell the runtime which kernel variants to select for which problem sizes.

Assumptions Embedded in the Message

The user's message makes several assumptions, most of which are well-founded. First, it assumes that kernel tuning is a meaningful lever for performance on this hardware. Given the 100% GPU utilization with only 55% power draw, this is a reasonable inference — the GPU is not power-limited, so better kernel configurations could increase computational throughput without hitting thermal or electrical constraints.

Second, the user assumes that the assistant has the tools and knowledge to perform kernel tuning. This is a reasonable assumption given that the assistant had already discovered flashinfer's autotuner infrastructure ([msg 284]), the gen_cutlass_fused_moe_sm120_module function ([msg 283]), and the tuning config format (<msg id=286-287>). The pieces were all there; the user was directing the assistant to assemble them.

Third, the user assumes that the performance bottleneck is kernel-related rather than architectural. This assumption turned out to be partially correct but incomplete. As the assistant would later discover ([msg 297]), the flashinfer_cutlass MoE path does not use the flashinfer autotuner at all — the autotuner is only for the flashinfer_trtllm path. The CUTLASS path uses a fixed kernel with tune_max_num_tokens as its only configurable parameter. So while the user's instinct was correct that the new GPU needed custom tuning, the specific tuning infrastructure they were pointing to did not apply to the backend that was actually working.

The Knowledge Landscape

To fully appreciate this message, one must understand the technical context. The model being deployed is GLM-5-NVFP4, a Mixture-of-Experts model with 256 experts, each with a hidden dimension of 256 (at TP8). The MoE computation involves routing each token through a subset of experts, each performing a GEMM (General Matrix Multiply) operation. These operations are small — batch size 1 during decode, hidden size 256 — which means they cannot saturate the GPU's tensor cores. The challenge is to minimize kernel launch overhead and maximize the efficiency of each small GEMM.

The GPU hardware is the RTX PRO 6000 Blackwell, an SM120 architecture GPU with 96 GB of memory and a 600W TDP. It is a workstation variant of the Blackwell architecture, distinct from the datacenter B200 (SM100). The flashinfer library provides SM120-specific kernel code but lacks the tuning configs that map problem dimensions to optimal kernel configurations for this specific chip.

The assistant had previously investigated MoE tuning configs from a prior project deploying Kimi K2 on the same hardware (<msg id=271-272>). Those configs were generated using tuning_fused_moe_triton.py and were specific to the Triton MoE runner backend with FP8 quantization — a completely different code path from the NVFP4 CUTLASS path being used here. The user's message implicitly references this prior work, suggesting that the same tuning approach should be applied to the current deployment.

What This Message Created

The user's directive set in motion a chain of investigation that would prove highly informative, even if it didn't yield the immediate performance gains one might hope for. The assistant proceeded to:

  1. Examine flashinfer's autotuner infrastructure in detail (<msg id=284-289>), discovering the AutoTuner class, the autotune context manager, and the config loading mechanism.
  2. Confirm that no tuning config existed for the RTX PRO 6000 ([msg 285]).
  3. Investigate whether the autotuner was even being invoked during server warmup (<msg id=293-296>), discovering that the flashinfer_cutlass path does not use the autotuner.
  4. Test the flashinfer_cutedsl backend as an alternative (<msg id=298-303>), finding comparable throughput (~206 tok/s output).
  5. Ultimately conclude that the MoE runner backend choice does not significantly change throughput on this hardware ([msg 304]). This last discovery was crucial: by testing all viable backends and finding them all within the same performance band (195-225 tok/s output), the assistant confirmed that the bottleneck was not in the MoE kernel implementation but elsewhere — likely in the attention mechanism, the all-reduce communication, or the fundamental small-batch decode geometry.

A Missed Opportunity and a Valuable Lesson

The user's message contains one incorrect assumption that is worth examining. The phrase "tune the kernels" implies that there exists a tuning mechanism that, when applied, will produce better kernel configurations. For the flashinfer_trtllm backend, this is true — the autotuner profiles multiple tactic choices at runtime and selects the fastest. But for the flashinfer_cutlass backend that was actually working, the kernels are compiled with fixed configurations and the only tuning parameter is tune_max_num_tokens, which controls how many tokens are batched into a single kernel invocation.

This is a subtle but important distinction. The user correctly identified that the new GPU needs custom configurations, but the infrastructure for providing those configurations (the flashinfer autotuner) only covers one of the five available MoE runner backends. The CUTLASS SM120 kernels are compiled from template code with architecture-specific optimizations baked in — they don't have a tuning config system in the same sense.

Nevertheless, the user's intervention was valuable. It forced a deeper investigation of the kernel infrastructure, revealed the architecture of flashinfer's autotuner, and ultimately led to the conclusion that the performance ceiling was not in the MoE kernels but in the overall system architecture. The message exemplifies a critical skill in ML engineering: knowing when to stop swapping components and start understanding why the components behave the way they do on your specific hardware.