The SM120 Check: A Pivotal Verification in the Blackwell Optimization Journey

Introduction

In the middle of an intensive optimization session targeting the GLM-5-NVFP4 large language model on NVIDIA's cutting-edge Blackwell GPUs, a single assistant message stands out as a critical turning point. Message 1241, consisting of a brief observation and a single bash command, represents the moment when a theoretical optimization path was validated as practically viable. The message reads:

Notice this line: if major == 11 and minor == 0: raise ValueError("SM110 is not supported"). SM120 (12.0) is not blocked. But let me check if there's an SM120-specific block elsewhere:

>

``bash ssh root@10.1.230.174 'source /root/ml-env/bin/activate && python3 -c " from flashinfer.cute_dsl.blockscaled_gemm import get_compute_capability import torch major, minor = get_compute_capability(torch.device(\"cuda:0\")) print(f\"Compute capability: {major}.{minor}\") " 2>&1' Compute capability: 12.0 ``

This seemingly simple exchange — noticing a code guard, inferring the absence of a block, and verifying the hardware capability — encapsulates the essence of the engineering detective work that characterized this entire optimization campaign. To understand why this message matters, we must trace the threads of reasoning, context, and motivation that led to it.

The Broader Context: A 30x Performance Gap

The optimization session had been running for many rounds across multiple segments, each tackling a different aspect of inference performance on the GLM-5-NVFP4 model. This model, a 5-billion-parameter Mixture-of-Experts (MoE) architecture quantized to NVFP4 (NVIDIA's 4-bit floating point format), was being deployed on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The Blackwell architecture, with its SM120 compute capability (12.0), represented the latest generation of NVIDIA hardware, but its very newness meant that software support was still catching up.

The central problem was stark: theoretical analysis had shown that the maximum possible single-stream throughput was approximately 309 tokens per second, but actual measured performance was a dismal 10.36 tokens per second — a staggering 3.4% efficiency gap. Every decode step was taking roughly 97 milliseconds when theory suggested it should take only about 3.2 milliseconds. The team had been systematically working through every possible bottleneck: CPU governor settings, kernel versions, NUMA balancing, PCIe configuration, and dozens of software optimizations.

One of the most promising avenues for improvement was the MoE (Mixture-of-Experts) kernel backend. The GLM-5-NVFP4 model's architecture relies heavily on MoE layers, where each token is routed to a subset of expert networks. The efficiency of the GEMM (General Matrix Multiply) operations within these MoE layers was the single largest contributor to the performance gap. SGLang, the inference serving framework being used, supported multiple MoE runner backends, including Triton, FlashInfer CUTLASS, and — most intriguingly — FlashInfer CuteDSL.

The CuteDSL Promise

The FlashInfer CuteDSL backend represented a potentially transformative optimization. CuteDSL (CUDA Template Expressions DSL) is a domain-specific language embedded in Python that allows for the generation of highly optimized CUDA kernels at JIT compile time. Unlike traditional GEMM implementations that use fixed tile sizes and memory layouts, CuteDSL can generate kernels that are specifically tuned for the exact shapes, data types, and hardware characteristics of the problem at hand.

For the GLM-5-NVFP4 model, the key challenge was that each MoE expert operates on very small matrices — a single token (M=1) multiplied by a weight matrix with inner dimension K=6144 and output dimension N=256. These "small GEMMs" are notoriously difficult to accelerate because they cannot fully utilize the GPU's massive parallel compute units. The FP4 quantization further complicates matters by requiring block-scaled matrix operations with non-standard data layouts.

The CuteDSL backend had been designed specifically to handle these kinds of challenging GEMM shapes, and it included support for the NVFP4 block-scaled format through the grouped_gemm_nt_masked function. If it worked on SM120, it could potentially double or triple the MoE kernel throughput.

The Critical Question: Does CuteDSL Support Blackwell?

This brings us to message 1241. In the preceding messages (1230–1240), the assistant had been exploring the FlashInfer CuteDSL codebase, checking whether the module could be imported, verifying that the grouped_gemm_nt_masked function existed, and examining the source code for architecture requirements. The assistant had found the CuteDSL module, confirmed it imported successfully, and was now looking at the architecture-specific code paths.

The message begins with a sharp observation: "Notice this line: if major == 11 and minor == 0: raise ValueError("SM110 is not supported")." This line was found in the CuteDSL source code, and the assistant immediately recognized its significance. The code explicitly blocks SM110 (the compute capability for NVIDIA's previous-generation Hopper architecture), but it does NOT block SM120 (Blackwell). This is a classic software archaeology technique: when you see an explicit exclusion, you can infer that everything else is at least not explicitly excluded.

However, the assistant was careful not to over-interpret this absence. The lack of an SM120 block could mean:

  1. SM120 is fully supported and tested
  2. SM120 support hasn't been added yet but will work by default
  3. SM120 hasn't been tested and might fail at runtime for other reasons The assistant chose to verify empirically by checking the compute capability of the actual GPU hardware. The command get_compute_capability(torch.device("cuda:0")) returns the major and minor version numbers of the CUDA compute capability for the first GPU. The result — 12.0 — confirmed that the hardware was indeed SM120.

The Thinking Process: What Made This Message Necessary

This message exists at the intersection of several lines of reasoning that had been developing over the preceding rounds:

First, the theoretical performance analysis had established that the GEMM operations in the MoE layers were the dominant bottleneck. The assistant had computed that weight reads alone should take only 1.59ms per token at peak bandwidth, but the actual decode step was taking 97ms. This 60x discrepancy pointed squarely at kernel efficiency.

Second, the exploration of MoE backends had revealed that the current flashinfer_cutlass backend, while functional, was not optimized for the specific GEMM shapes and data types of the GLM-5-NVFP4 model. The CuteDSL backend promised JIT-compiled kernels tuned for exactly these parameters.

Third, the hardware constraint — SM120 compute capability — was a potential blocker. Blackwell GPUs were still new enough that many CUDA libraries had not yet added full support. The assistant had to determine whether CuteDSL would work on these GPUs before investing time in configuring and launching a server with the new backend.

The decision to run a simple Python verification command rather than immediately launching a full server test reflects sound engineering judgment. A failed server launch would waste 5-10 minutes of startup time and potentially leave the system in an inconsistent state. A quick Python import and capability check, by contrast, takes seconds and provides clear yes/no information about the fundamental compatibility question.

Assumptions and Potential Pitfalls

The assistant made several assumptions in this message, most of which were reasonable but worth examining:

Assumption 1: The absence of an SM120 block implies support. This is a logical inference but not a guarantee. The code might simply not have been updated to block SM120 yet, and runtime failures could still occur due to missing PTX instructions, incorrect warp sizes, or other architecture-specific details that aren't captured by a simple version check.

Assumption 2: get_compute_capability returns the correct value. This is a safe assumption — this function is part of the FlashInfer library and directly queries the CUDA runtime for the device's compute capability. The result 12.0 is consistent with the known specifications of the RTX PRO 6000 Blackwell GPU.

Assumption 3: The Python environment has the correct CUDA toolkit and driver versions. The command runs inside the /root/ml-env/bin/activate virtual environment, which had been carefully set up with CUDA Toolkit 13.1 and compatible PyTorch and FlashInfer versions. If the environment were misconfigured, the capability check might return incorrect results or fail entirely.

Assumption 4: All GPUs have the same compute capability. The check only queries cuda:0. In a multi-GPU system, it's possible (though extremely unlikely in this homogeneous setup) that different GPUs have different capabilities. The assistant implicitly assumes homogeneity, which is reasonable for a system with 8 identical RTX PRO 6000 GPUs.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of CUDA compute capabilities: Understanding that SM120 refers to the Blackwell architecture (compute capability 12.0) and that different GPU generations have different compute capabilities that affect which kernels can run.
  2. Familiarity with the GLM-5-NVFP4 model architecture: Understanding that this is an MoE model quantized to 4-bit floating point, and that the MoE layers are the primary computational bottleneck during inference.
  3. Knowledge of the FlashInfer library: Understanding that FlashInfer provides multiple MoE kernel backends, that CuteDSL is a JIT-compilation approach for generating optimized kernels, and that the grouped_gemm_nt_masked function handles block-scaled FP4 GEMMs.
  4. Context from the ongoing optimization session: The preceding 30+ messages had established the performance gap, explored various backends, and narrowed the focus to CuteDSL as a promising optimization path.
  5. Understanding of the SGLang inference framework: Knowing that SGLang supports multiple MoE runner backends and that the --moe-runner-backend flag controls which backend is used.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

  1. Confirmed SM120 compatibility: The FlashInfer CuteDSL module's get_compute_capability function correctly identifies the Blackwell GPU as SM120, and the absence of an SM120 block in the architecture guard suggests (though doesn't guarantee) that the CuteDSL kernels will compile and run on this hardware.
  2. Validated the verification approach: The assistant demonstrated a lightweight, low-risk method for checking hardware compatibility before committing to a full server reconfiguration. This pattern — check first, then deploy — is a best practice for production systems.
  3. Created a decision point: With SM120 compatibility confirmed, the next logical step was to launch a test server with --moe-runner-backend flashinfer_cutedsl and benchmark the results. The message thus serves as the bridge between exploration and action.
  4. Documented the reasoning process: The message captures the assistant's thinking — noticing a code pattern, drawing an inference, and then empirically verifying. This kind of reasoning trace is invaluable for understanding how engineering decisions are made in complex optimization scenarios.

The Significance of Small Verifications

In the context of a large-scale optimization effort spanning dozens of messages and multiple days of work, message 1241 might seem minor. It's a single observation and a single command. But this is precisely the kind of verification that separates successful optimization from guesswork.

The assistant could have simply assumed that CuteDSL would work on SM120 and proceeded to launch a server. If that assumption had been wrong, the server would have failed to start, wasting time and potentially leaving the system in a broken state. Alternatively, the assistant could have assumed CuteDSL wouldn't work and abandoned the line of inquiry entirely, missing out on a potentially transformative optimization.

Instead, the assistant followed the scientific method: observe a clue (the SM110 block), form a hypothesis (SM120 is not blocked, therefore likely supported), and test the hypothesis (query the compute capability). This approach minimizes wasted effort while systematically exploring the optimization space.

What Came Next

The verification in message 1241 opened the door to testing the CuteDSL backend. In subsequent messages, the assistant would launch a server with the new backend, run benchmarks, and measure the actual throughput improvement. The results would feed back into the ongoing analysis, informing decisions about which optimizations to pursue further and which to abandon.

The broader lesson is that in complex systems optimization, the most valuable work often happens at the boundaries — checking compatibility, verifying assumptions, and confirming that a promising path is actually viable before committing resources. Message 1241 exemplifies this principle, showing how a moment of careful observation and quick verification can unlock an entire line of optimization.