The Diagnostic Pivot: Deciphering sgl-kernel's Architecture Loading on SM120

In the middle of a high-stakes deployment of the Kimi-K2.5 model on 8× RTX PRO 6000 Blackwell GPUs, a single message marks the critical turning point where raw effort gives way to systematic investigation. After investing 48 minutes building the sgl-kernel CUDA library with explicit SM120 targeting, the assistant discovers that the build produced nothing for the target architecture — and instead of blindly rebuilding, pauses to understand why.

The Message

The subject message ([msg 3121]) is brief but decisive:

Still only SM90 and SM100 — no SM120. The build system doesn't produce an SM120-specific variant. Let me check the load_utils to understand the fallback logic:

``bash ssh root@10.1.230.174 'grep -n "sm120\|SM120\|120\|fallback\|Expected variant" /root/ml-env/lib/python3.12/site-packages/sgl_kernel/load_utils.py | head -20' 108: # Continue to fallback 118: logger.debug(f"[sgl_kernel] Attempting fallback: looking for pattern {alt_pattern}") 119: logger.debug(f"[sgl_kernel] Found fallback files: {raw_alt_files}") 120: logger.debug(f"[sgl_kernel] Prioritized fallback files: {alt_matching_files}") 124: logger.debug(f"[sgl_kernel] Found fallback library: {alt_path}") 134: logger.debug(f"[sgl_kernel] Loading fallback module from {alt_path}...") 136: logger.debug(f"[sgl_ker... ``

The Road That Led Here

To understand the weight of this moment, we must trace the path that preceded it. The assistant had been working for hours — across multiple sessions spanning days — to deploy speculative decoding for the 1-trillion-parameter Kimi-K2.5 model. The journey had been grueling: a complete EAGLE-3 training pipeline was built from scratch, 10,000 synthetic samples were generated over 5.3 hours, 828 GB of hidden states were extracted, and a drafter model was finetuned over 5 epochs in 2.6 hours.

Then came the vLLM integration. Despite three custom patches to make vLLM's EAGLE-3 implementation work with the DeepSeek V3/Kimi-K2.5 architecture (model whitelist, image token handling, and the SupportsEagle3 interface), both the newly trained drafter and the pre-trained AQ-MedAI baseline achieved only ~15% acceptance rate — resulting in 0.66× throughput, worse than no speculation at all. The root cause was identified as a fundamental issue with how vLLM extracts auxiliary hidden states from MLA (Multi-head Latent Attention) layers during decode.

The user's directive was clear: pivot to SGLang, which has first-class EAGLE-3 support and is explicitly tested with Kimi-K2 drafters, achieving reported 1.8× speedups.

The Build That Didn't Build

The pivot began with installing sgl-kernel, the CUDA kernel library underpinning SGLang. The source code at /root/sglang/sgl-kernel/ already contained SM120 support in its CMake configuration — the CMakeLists.txt included the flag -gencode=arch=compute_120a,code=sm_120a. The assistant set SGL_KERNEL_CUDA_ARCHS="120" and launched the build.

The build completed successfully in 48 minutes and 32 seconds. The output declared:

Built sgl-kernel @ file:///root/sglang/sgl-kernel
Prepared 1 package in 48m 32s
Installed 1 package in 20ms

But when the assistant tried to import sgl_kernel ([msg 3116]), it crashed with:

ImportError: [sgl_kernel] CRITICAL: Could not load any common_ops library!

The first diagnostic step ([msg 3117]) revealed that the editable install (uv pip install -e) hadn't produced any .so files at all in the python directory. A second attempt with a non-editable install ([msg 3119]) also reported success in 48 minutes, but inspection ([msg 3120]) showed only sm90/common_ops.abi3.so and sm100/common_ops.abi3.so — no sm120/ directory existed.

The Reasoning in the Message

This is where message 3121 enters. The assistant makes a critical observation: "Still only SM90 and SM100 — no SM120. The build system doesn't produce an SM120-specific variant."

This sentence contains two distinct claims. The first is a factual observation: the file system shows only two architecture directories. The second is a hypothesis: that the build system is incapable of producing an SM120 variant. The assistant is careful not to treat this as proven fact — the very next sentence proposes an investigation: "Let me check the load_utils to understand the fallback logic."

The decision to examine load_utils.py is a masterclass in debugging methodology. Rather than continuing to tweak build flags, re-run the build, or search for documentation, the assistant goes straight to the code that determines which binary to load at runtime. This is the correct level of abstraction: the build may have produced SM120 code, but if the loading logic doesn't know where to find it, the result is indistinguishable from a build failure.

The grep command is carefully crafted. It searches for five patterns simultaneously:

sm120 and SM120 — to find explicit SM120 handling
120 — as a broader net for any reference to the compute capability number
fallback — to understand the fallback mechanism that might save the day
Expected variant — to find error messages that reveal the expected directory structure The output reveals the fallback system's structure: line 108 begins the fallback logic, lines 118-120 handle pattern matching for alternative library files, line 124 reports finding a fallback library, and lines 134-136 handle loading it. Notably absent from the grep output is any explicit sm120 or SM120 reference — confirming that SM120 is not a first-class architecture in the loading code.

Assumptions and Their Consequences

Several assumptions underpin this message, some of which would prove incorrect:

Assumption 1: The build system doesn't produce SM120-specific variants. This was the assistant's working hypothesis, and it drove the investigation toward the loading logic. In reality, as later messages would reveal ([msg 3125]), the CMake build did compile for SM120 — the binary in sm100/common_ops.abi3.so contained SM120 code thanks to the -gencode=arch=compute_120a,code=sm_120a flag. The problem was that load_utils.py mapped compute capability 120 to the sm100 directory, and the binary required libtorch.so to be in the library path.

Assumption 2: The SGL_KERNEL_CUDA_ARCHS environment variable controls architecture targeting. The assistant set this to "120" expecting only SM120 code to be produced. However, the CMake configuration independently specifies architectures, and the env var may not have been propagated correctly through the build system.

Assumption 3: The editable install compiled CUDA code. The first build attempt used -e (editable mode), which with scikit-build-core often creates a pointer to the source tree rather than fully compiling and installing. This explained the complete absence of .so files after the first build.

Assumption 4: A successful build message means a usable build. The build system reported "Built sgl-kernel" without error, but the runtime behavior was broken. This is a common pitfall with CUDA builds: the compilation may succeed but produce binaries for the wrong architectures, or the installation step may not place files where the loader expects them.

Input Knowledge Required

Understanding this message requires substantial domain knowledge:

CUDA compute capabilities: The reader must know that sm90 corresponds to Hopper (H100), sm100 to first-generation Blackwell, and sm120 to the next-generation Blackwell architecture found in RTX PRO 6000 GPUs. The mapping from compute capability number to directory name is not obvious — capability 120 maps to directory sm100, not sm120.
sgl-kernel's architecture-specific loading: The library uses a directory-per-architecture scheme (sm90/, sm100/) with a common_ops.abi3.so shared object in each. The load_utils.py module detects the GPU's compute capability at import time and selects the appropriate subdirectory.
The fallback mechanism: The grep output reveals that when the primary architecture directory is missing, the loader attempts fallback — searching for alternative library files that might be compatible. This is crucial for forward compatibility: an SM120 GPU might be able to load SM100 binaries if they were compiled with compatible instruction sets.
The broader deployment context: This message sits within a multi-day effort to deploy Kimi-K2.5 with speculative decoding. The failure of vLLM's EAGLE-3 integration ([msg 3089]) had already cost significant time, and the pivot to SGLang represented a bet on a different implementation. Every hour spent debugging sgl-kernel was time not spent testing the actual speculative decoding performance.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

The fallback system's structure: The grep output reveals the line numbers and function structure of the fallback logic. Lines 108-136 of load_utils.py contain the fallback mechanism, with specific debug log messages that can be enabled for further diagnostics.
The absence of SM120 as a first-class architecture: The grep found no explicit sm120 or SM120 references, confirming that SM120 support would require either adding a new directory or making the fallback mechanism work.
A debugging strategy: The assistant demonstrates a methodology — when a build succeeds but runtime fails, investigate the loading code, not the build code. This insight is transferable to any similar situation.
The next investigative step: The grep output points directly at the fallback mechanism as the most promising path forward. If the SM100 binary can serve as a fallback for SM120 (forward compatibility), the loading issue can be resolved without rebuilding.

The Thinking Process

The assistant's reasoning in this message follows a clear arc:

Observation: The file system inspection shows only sm90 and sm100 directories.
Hypothesis formation: "The build system doesn't produce an SM120-specific variant" — this is a claim about the build system's behavior, not about the hardware or the loading code.
Hypothesis testing strategy: Instead of continuing to tweak build parameters, investigate the loading code. This is a higher-leverage approach: if the loading code is the bottleneck, fixing the build won't help.
Tool selection: The grep command is chosen to extract all relevant lines from the loading code in one shot, with multiple search patterns to catch both explicit architecture references and fallback-related logic.
Result interpretation: The grep output shows the fallback mechanism's structure but no SM120-specific handling. This confirms that SM120 isn't a first-class architecture in the loader, but leaves open the possibility that the fallback mechanism could provide compatibility. The thinking is notably systematic. The assistant doesn't panic, doesn't restart the build, and doesn't ask the user for guidance. It follows the evidence: the build succeeded, the files exist, but the loader can't find them. Therefore, the problem is in the loader, not the build.

The Broader Significance

This message represents a classic debugging pattern in machine learning infrastructure: the "successful build, failed runtime" problem. When working with bleeding-edge hardware (SM120 Blackwell GPUs), the build system and the runtime system often have different understandings of what "support" means. The build system may compile code for an architecture, but if the runtime loader doesn't know to look for it, or if the binary is placed in the wrong directory, the result is indistinguishable from no support at all.

The assistant's decision to investigate load_utils.py rather than the build configuration is what ultimately leads to the solution. In subsequent messages ([msg 3122]), the assistant reads the full loading logic and discovers that SM120 maps to the sm100 directory. In [msg 3123], it attempts to load the SM100 binary directly and encounters a missing libtorch.so error — a separate issue. And in [msg 3126], after ensuring torch is imported first (which loads libtorch.so into the process), sgl_kernel imports successfully.

The chain of reasoning that begins in message 3121 — "the build system doesn't produce an SM120-specific variant, let me check the load_utils" — is the critical diagnostic pivot that turns a seemingly intractable build failure into a solvable loading-path issue. It is a textbook example of debugging at the right level of abstraction.