The Missing Config: Discovering FlashInfer's Autotuner Gap for Blackwell RTX PRO 6000

In the middle of a marathon debugging session to deploy the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant made a discovery that would redefine the trajectory of the optimization effort. Message [msg 285] is a brief but pivotal moment — two bash commands, two lines of output, and a quiet revelation that the flashinfer library had never seen this GPU before. The message reads:

The autotuner looks for device-specific config files at flashinfer/tuning_configs/v0_1_trtllm_fused_moe_<device_name>.py. Let me check if one exists for our GPU:

[bash] ssh 10.1.230.175 'source ~/ml-env/bin/activate && python3 -c " from flashinfer.autotuner import get_config_path print(\"Module path:\", get_config_path(True)) print(\"File path:\", get_config_path(False)) import os print(\"File exists:\", os.path.exists(get_config_path(False))) " 2>&1' Module path: flashinfer.tuning_configs.v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition File path: /home/theuser/ml-env/lib/python3.12/site-packages/flashinfer/tuning_configs/v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.py File exists: False

[bash] ssh 10.1.230.175 'ls /home/theuser/ml-env/lib/python3.12/site-packages/flashinfer/tuning_configs/ 2>/dev/null || echo "No tuning_configs dir"' v0_1_trtllm_fused_moe_NVIDIA_B200.py v0_1_trtllm_fused_moe_NVIDIA_GB200.py

This message is the culmination of a chain of reasoning that began with a frustrating performance observation. Earlier in the session, the assistant had benchmarked the GLM-5-NVFP4 model and found that all eight GPUs were running at 100% utilization but only drawing 55% of their 600W power budget ([msg 269]). The GPUs were compute-bound yet underpowered — a paradox that suggested the MoE (Mixture-of-Experts) kernels were not efficiently using the tensor cores. The user's directive at [msg 280] — "Also probaby tune the kernels, this is really new gpu" — crystallized the hypothesis: the RTX PRO 6000 Blackwell (compute capability SM120) was so new that no pre-tuned kernel configurations existed for it.

The Chain of Discovery

The assistant did not arrive at this discovery by accident. It was the product of a methodical, multi-step investigation that traced through the flashinfer library's architecture. In the preceding messages, the assistant had already:

Identified the MoE runner backend: The server was using flashinfer_cutlass, which employs a CUTLASS-based FP4 fused MoE kernel path ([msg 274]).
Discovered SM120-specific code: A Python introspection of flashinfer's fused_moe module revealed gen_cutlass_fused_moe_sm120_module — a function that generates SM120-specific CUTLASS MoE modules ([msg 283]). This confirmed that flashinfer did have code paths for this GPU architecture, but they might not be optimally configured.
Found the autotuner: A file search across the flashinfer package directory turned up autotuner.py, along with SM120-specific CUDA source files like fp4_gemm_cutlass_sm120.cu and fp4_gemm_template_sm120.h ([msg 284]). The autotuner's get_config_path function was the key — it constructs a file path based on the GPU's device name. The subject message executes the final verification: it calls get_config_path to discover the expected config file path, checks whether the file exists, and lists the tuning_configs directory to see what does exist. The result is unambiguous: the expected file v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition.py does not exist. Only configs for the NVIDIA B200 (SM100, datacenter Blackwell) and NVIDIA GB200 (Grace-Hopper Blackwell) are present.

Why This Matters

The absence of a tuning config file has concrete performance implications. As the assistant would discover in the following message ([msg 287]), when the config file doesn't exist, the autotuner's load_from_file function returns (False, 0, -1, None) — falling back to default config 0 with tile_config -1. This means every MoE kernel invocation uses suboptimal tile sizes and scheduling parameters for the SM120 architecture.

The RTX PRO 6000 Blackwell is a workstation GPU with 96GB of memory and a 600W TDP, built on the same Blackwell architecture as the datacenter B200 but with different compute capabilities (SM120 vs SM100). The flashinfer library's tuning configs encode hardware-specific CUTLASS tile configurations — parameters like block sizes, warp counts, and pipeline stages that determine how efficiently the GPU's tensor cores execute matrix multiplications. A config tuned for the B200's SM100 architecture may select tile sizes that are suboptimal for the SM120's different register file size, shared memory capacity, or warp scheduling characteristics.

The Broader Context: A New GPU on the Block

This discovery sits within a larger narrative about deploying cutting-edge hardware. The RTX PRO 6000 Blackwell was released as part of NVIDIA's workstation lineup, and the flashinfer library — a critical dependency for efficient MoE inference in SGLang — had clearly been tuned primarily for datacenter Blackwell variants (B200, GB200). The RTX PRO 6000, with its SM120 compute capability, was an afterthought.

The assistant's investigation reveals an implicit assumption baked into the flashinfer library: that the Blackwell GPUs that matter are the datacenter ones. This is a reasonable assumption for a library focused on production inference serving, but it creates a gap for researchers and engineers deploying on workstation Blackwell hardware. The two existing config files — v0_1_trtllm_fused_moe_NVIDIA_B200.py and v0_1_trtllm_fused_moe_NVIDIA_GB200.py — represent the configurations that NVIDIA's TensorRT-LLM team deemed important enough to tune.

What This Message Creates

The output knowledge from this message is twofold. First, it provides a definitive answer to the question "does flashinfer have a pre-tuned config for this GPU?" — no. Second, it establishes the exact file path and naming convention that would be needed to create such a config. The module path flashinfer.tuning_configs.v0_1_trtllm_fused_moe_NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition tells us exactly what Python module would need to exist, and the existence of the B200 config provides a template for the format.

This knowledge sets up the next phase of the investigation: understanding the tuning config format, determining how to run the flashinfer autotuner to generate a config, and ultimately either running the autotuner on the target hardware or manually crafting config entries for the GLM-5-NVFP4 model's specific dimensions. The message is a turning point — it transforms the optimization problem from "tweak parameters" to "generate the missing tuning data."

The Thinking Process

The assistant's reasoning in this message reveals a systematic approach to debugging performance issues in deep learning infrastructure. Rather than guessing at parameters or trying random combinations of flags, the assistant traces through the library's source code to understand the mechanism by which kernel configurations are selected. The use of Python introspection (get_config_path, os.path.exists) and shell commands (ls) is efficient and precise — each command answers a specific question.

The assistant also demonstrates an understanding of the flashinfer library's architecture: that it uses device-specific tuning configs, that these configs are Python files with a specific naming convention, and that the autotuner falls back to defaults when no config exists. This knowledge comes from the preceding investigation (<msg id=282-284>) where the assistant explored the autotuner's API surface.

In many ways, this message exemplifies the kind of detective work required when deploying models on novel hardware. The GPU is new, the library hasn't been tuned for it, and the performance is suboptimal. The first step to fixing the problem is understanding why the performance is suboptimal — and the assistant correctly identifies the missing tuning config as a root cause. The message doesn't solve the problem, but it defines it precisely, which is often the harder and more valuable step.