Diagnosing Architecture Mismatch: The SM120 vs SM100 Kernel Boundary

In the high-stakes world of deploying large language models on cutting-edge hardware, the difference between a working configuration and a crashing one often comes down to a single digit in a compute capability version number. Message [msg 5962] captures precisely such a moment — a brief but analytically dense diagnostic turn in a broader effort to deploy the Qwen3.5-397B-A17B-NVFP4 model on an 8× RTX PRO 6000 Blackwell GPU system using SGLang. The message is deceptively short, containing only two bash commands and a few lines of reasoning, but it represents a critical juncture where the assistant correctly interprets a crash, identifies the root cause at the level of CUDA architecture targeting, and pivots to an alternative backend strategy.

The Context: Systematic Backend Exploration

To understand why message [msg 5962] was written, we must trace the arc of the preceding investigation. The assistant had been working through Segment 39 of a long-running optimization session, having recently upgraded the entire software stack to nightly builds — PyTorch 2.12.0.dev20260307+cu130, flashinfer 0.6.5, and the latest SGLang main branch. The hardware target was the RTX PRO 6000 Blackwell GPU, which reports compute capability SM120 (major version 12, minor version 0). This is a brand-new architecture, and much of the GPU software ecosystem was still catching up.

The assistant had already built sgl-kernel from source with SM120 support by applying community patches from "catid" for CMake policy guards, CUDA 13 cccl include paths, and FA3 fallback. But the question of which inference backends actually worked correctly on SM120 remained open. The assistant had been methodically testing combinations of MoE runner backends and FP4 GEMM backends, cycling through flashinfer_cutlass, flashinfer_trtllm, flashinfer_cudnn, and flashinfer_cutedsl to find a configuration that produced both correct output and good performance.

In the messages immediately preceding [msg 5962] ([msg 5957] through [msg 5961]), the assistant had stopped the production server, cleaned the GPUs, and launched a test with --moe-runner-backend flashinfer_trtllm. The process was killed almost immediately — the shell output showed "Killed" with PID 43621. The assistant initially suspected an OOM condition ("probably OOM since trtllm requires weight shuffling which temporarily doubles memory"), but then checked the log file to find the real cause.

The Diagnostic Insight

Message [msg 5962] opens with the assistant's analysis of the crash log from the flashinfer_trtllm attempt:

As expected — trtllm_fp4_block_scale_moe tries to compile for SM100 (major version 10) and our GPUs are SM120 (major version 12). So flashinfer_trtllm does NOT work on SM120 for MoE. The fused TRT-LLM MoE kernels are SM100-only.

This is the core diagnostic insight. The TRT-LLM (TensorRT-LLM) fused MoE kernel, exposed through flashinfer as trtllm_fp4_block_scale_moe, was compiled with architecture guards targeting SM100 — the compute capability of NVIDIA's previous-generation Hopper/H100 architecture (compute capability 10.0). The Blackwell RTX PRO 6000 GPUs report SM120, and the kernel simply refuses to load or compile on that architecture. The "Killed" signal was not an OOM condition but rather a fatal error during kernel compilation or loading, likely an assertion failure or architecture check deep inside the TRT-LLM code path.

The assistant's phrasing — "As expected" — is telling. This was not a surprise. The assistant had been probing the codebase for architecture compatibility throughout the session. Earlier, in [msg 5947], the assistant had checked which files in the MoE runner directory referenced is_sm120 or is_sm100, finding that flashinfer_trtllm.py imported is_sm120_supported but only used it as a guard for fp4_quantize imports, not for the fused MoE kernel itself. The crash confirmed what the code inspection had hinted at: the fused TRT-LLM MoE path was SM100-only, and no amount of patching would make it run on SM120 without significant kernel recompilation.

The Pivot to flashinfer_cutedsl

Having eliminated flashinfer_trtllm for the MoE runner, the assistant immediately pivoted to the next candidate:

Let me try flashinfer_cutedsl instead — it should JIT-compile for SM120:

This is a reasoned decision based on architectural knowledge. The cutedsl backend uses NVIDIA's CUTE (CUDA Templates) DSL, which generates CUDA kernels at runtime through JIT compilation. Unlike the pre-compiled TRT-LLM kernels, which are baked for specific architectures, a JIT-compiled approach should theoretically adapt to whatever compute capability it detects at runtime. The assistant's reasoning is sound: if cutedsl truly JIT-compiles, it should work on SM120.

The assistant then executes two bash commands. The first forcefully terminates any lingering processes on the NVIDIA devices and confirms the GPUs are clean:

fuser -k /dev/nvidia* 2>/dev/null; sleep 2; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader | head -2

The output shows all GPUs at 0 MiB — clean state, ready for the next experiment. This cleanup step is crucial because a failed server launch can leave GPU memory allocated, and subsequent launches would fail with CUDA out-of-memory errors.

Assumptions Embedded in the Message

The assistant makes several assumptions in this message, most of which are reasonable but not all of which hold up under further testing.

The primary assumption is that flashinfer_cutedsl will produce correct output on SM120 simply because it can JIT-compile. The assistant's reasoning focuses on compatibility — will the kernel load and run? — rather than correctness — will it produce the right numerical results? This is a natural division of labor: first get it running, then verify correctness. But the assumption that JIT-compilation guarantees correctness is fragile. CUTE DSL kernels, like any templated code, can compile successfully on an unsupported architecture but produce incorrect results due to subtle differences in instruction behavior, register file size, or warp scheduling.

A secondary assumption is that the architecture mismatch is the only reason flashinfer_trtllm failed. The assistant does not investigate whether there might be additional issues — memory layout differences, scale factor format incompatibilities, or routing logic mismatches — that would also need fixing. This is a pragmatic choice: once the root cause is identified (SM100-only kernel), there is little value in deeper investigation of a dead-end path.

The assistant also assumes that flashinfer_cutedsl is a drop-in replacement that will work with the same model weights and configuration. This is reasonable given that all the flashinfer backends share the same interface (mm_fp4 function with a backend parameter), but different backends may have different requirements for weight layout, scale factor arrangement, or shuffle patterns.

Input Knowledge Required

To fully understand message [msg 5962], the reader needs substantial background knowledge spanning several domains.

First, one must understand CUDA compute capabilities — the SM120 vs SM100 distinction. NVIDIA GPUs are identified by a compute capability version number where the major version indicates the architecture generation: SM80 for Ampere, SM90 for Ada Lovelace, SM100 for Hopper, SM120 for Blackwell. The trtllm_fp4_block_scale_moe kernel was compiled with architecture-specific code paths that only activate for SM100, and the kernel's compilation guards reject SM120 outright.

Second, one needs familiarity with the flashinfer backend architecture. Flashinfer provides multiple backends for the same mathematical operation (FP4 matrix multiplication and MoE routing), each implemented with different kernel technologies: TRT-LLM (TensorRT-LLM fused kernels), CUTLASS (NVIDIA's templated linear algebra library), cuDNN (NVIDIA's deep neural network library), and CUTE DSL (CUTE Domain-Specific Language for JIT-compiled kernels). The assistant's decision to try cutedsl next reflects knowledge that JIT-compiled kernels are more likely to adapt to novel architectures.

Third, the reader needs to understand the SGLang server architecture and its command-line flags. The --moe-runner-backend flag selects the kernel implementation for the Mixture-of-Experts layers, while --fp4-gemm-backend selects the kernel for dense FP4 matrix multiplications. These are independent axes of configuration, and the assistant is systematically exploring the combinatorial space.

Fourth, one must grasp the concept of weight shuffling in FP4 quantization. The assistant's initial hypothesis that the crash was OOM-related ("trtllm requires weight shuffling which temporarily doubles memory") reflects knowledge that some backends require the model weights to be rearranged into a specific layout before execution, which can temporarily require double the memory budget.

Output Knowledge Created

Message [msg 5962] produces several pieces of actionable knowledge.

The primary output is the confirmed incompatibility of flashinfer_trtllm for MoE on SM120 Blackwell GPUs. This is not a fixable configuration issue — it is a fundamental architectural limitation. The fused TRT-LLM MoE kernels are compiled exclusively for SM100 and will not load on SM120. This knowledge saves future experimentation time: anyone deploying on Blackwell GPUs should not waste effort trying to use flashinfer_trtllm for the MoE runner.

The secondary output is the decision to test flashinfer_cutedsl as the next candidate, based on its JIT-compilation property. This sets the direction for the immediately following messages ([msg 5963] through [msg 5967]), where the assistant launches a server with flashinfer_cutedsl, finds that it loads successfully, but then discovers through a smoke test that it produces garbage output (the !!!! pattern). That discovery, in turn, leads to the final conclusion that only flashinfer_cutlass works correctly on SM120 — a finding that shapes the production deployment.

The message also implicitly documents the methodology for diagnosing backend failures on novel GPU architectures: (1) attempt to launch with the target backend, (2) inspect crash logs for architecture-specific error messages, (3) identify the root cause at the level of CUDA compute capability guards, (4) pivot to an alternative backend with a different compilation model, and (5) verify both compatibility and correctness.

The Thinking Process

The reasoning visible in message [msg 5962] follows a clear diagnostic pattern. The assistant begins with a hypothesis confirmation ("As expected"), indicating that the crash aligned with prior code inspection. The assistant had already seen in [msg 5947] that flashinfer_trtllm.py imported is_sm120_supported but only for a specific sub-feature, and the fused MoE kernel itself had no SM120 path. The crash validated this reading.

The assistant then articulates the root cause at two levels: the specific function (trtllm_fp4_block_scale_moe), the architecture mismatch (SM100 vs SM120), and the generalization ("The fused TRT-LLM MoE kernels are SM100-only"). This layered explanation — from specific symptom to general principle — is characteristic of strong diagnostic reasoning.

The pivot to flashinfer_cutedsl is justified with a specific technical rationale ("it should JIT-compile for SM120"). This is not a random guess but a reasoned selection based on the compilation model of each backend. The assistant is effectively using the cause of the failure (architecture-specific pre-compilation) to select the next candidate (JIT-compilation, which avoids pre-compilation).

The cleanup commands that follow are not mere housekeeping — they reflect an understanding that failed server launches can leave the GPUs in an inconsistent state, and that a clean launch requires confirmed zero memory usage across all devices. The fuser -k command forcefully terminates any process holding NVIDIA file handles, and the nvidia-smi query confirms the clean state.

The Broader Significance

Message [msg 5962] is a microcosm of the challenges faced when deploying cutting-edge AI models on cutting-edge hardware. The software ecosystem — SGLang, flashinfer, PyTorch, TRT-LLM — is in constant flux, with each component adding support for new architectures at different rates. The TRT-LLM fused MoE kernels, developed and optimized for Hopper (SM100), had not yet been ported to Blackwell (SM120) at the time of this session. The assistant's task was to navigate this uneven support matrix and find a working combination through systematic testing and informed reasoning.

The message also illustrates the importance of understanding the compilation model of different kernel backends. Pre-compiled kernels (TRT-LLM) offer peak performance but are tied to specific architectures. JIT-compiled kernels (CUTE DSL) offer broader compatibility but may produce incorrect results on untested architectures. Template-based kernels (CUTLASS) offer a middle ground — they are compiled at install time but use architecture-agnostic templates that can be instantiated for any compute capability. The assistant's eventual discovery that flashinfer_cutlass works correctly on SM120 while flashinfer_cutedsl produces garbage reflects these subtle differences in compilation and code generation.

In the end, message [msg 5962] is a testament to the value of reading crash logs carefully, understanding the architecture versioning scheme, and using that understanding to make informed decisions about which path to pursue next. The message is brief — barely a paragraph of reasoning and two bash commands — but it encapsulates a diagnostic process that separates effective deployment engineering from trial-and-error guesswork.