The Moment of Discovery: Unraveling an SM120 Build Mystery in SGLang's sgl-kernel

In the high-stakes world of deploying large language models on cutting-edge hardware, few moments are as simultaneously frustrating and illuminating as the one captured in message 3125 of this opencode session. The assistant, having just completed a 48-minute rebuild of SGLang's sgl-kernel library for SM120 (Blackwell) GPUs, discovers that the build has silently ignored its explicit architecture directive. This message is a turning point — a moment of diagnostic clarity where the assistant pivots from assuming the build failed to investigating why it built the wrong thing.

The Context: A Desperate Pivot to SGLang

To understand why this message matters, we must first appreciate the pressure that led to it. The session had been a grueling multi-day effort to deploy speculative decoding for the Kimi-K2.5 model — a 1-trillion-parameter giant — on a machine with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had painstakingly built a complete EAGLE-3 training pipeline, generated 828 GB of synthetic training data, and fine-tuned a draft model. Yet when integrated with vLLM, the results were devastating: only ~15% acceptance rate and 0.66× throughput — worse than running without speculation. The user's directive was unambiguous: pivot to SGLang, which has first-class EAGLE-3 support and is explicitly tested with Kimi-K2 drafters.

The pivot began in earnest. SGLang's base model loading was astonishingly fast — 22 seconds versus vLLM's 25 minutes — but the server processes were deadlocking after weight loading, with zero CPU or GPU utilization and no listening port. The culprit appeared to be the sgl-kernel library, which had been installed for SM100 (Hopper) but not SM120 (Blackwell). The assistant set out to rebuild it.

The 48-Minute Build and Its Aftermath

Message 3125 opens with the assistant stating a crucial observation: "It was rebuilt (22:32 = just now). But CMake built it for SM100, not SM120. The SGL_KERNEL_CUDA_ARCHS="120" env var wasn't being respected." This single sentence encapsulates the entire drama of the preceding hour. The assistant had spent nearly an hour compiling CUDA kernels, only to find that the environment variable intended to restrict compilation to SM120 had been silently ignored.

The decision to check the CMakeLists.txt is the key reasoning move here. Rather than continuing to tweak build flags or try different installation methods, the assistant steps back to examine the build system itself. The command is a targeted grep:

grep -r "CUDA_ARCHITECTURES\|cuda_archs\|SGL_KERNEL_CUDA\|sm_1" /root/sglang/sgl-kernel/CMakeLists.txt | head -20

This is a textbook debugging pattern: when an environment variable doesn't produce the expected effect, check whether the build system actually reads it. The assistant is operating under the assumption that SGL_KERNEL_CUDA_ARCHS is a recognized configuration parameter — an assumption that turns out to be incorrect, or at least incomplete.

What the Grep Reveals

The output is revealing. The CMakeLists.txt contains explicit CUDA architecture flags:

"-gencode=arch=compute_100a,code=sm_100a"
"-gencode=arch=compute_120a,code=sm_120a"

This is a critical finding. The CMake build system already includes SM120 support. It's not that the build system can't compile for Blackwell — it's that it compiles for both SM100 and SM120 simultaneously, and the SGL_KERNEL_CUDA_ARCHS variable doesn't override this. The assistant's earlier assumption that setting this variable would restrict the build to SM120 alone was mistaken.

This moment of discovery reshapes the entire debugging trajectory. The problem is no longer "the build didn't include SM120" but rather "the build included SM120 code but the runtime loader can't find it." The assistant's mental model shifts from a build-system problem to a runtime-loading problem.

Input Knowledge Required

To fully understand this message, the reader needs several layers of context:

  1. The hardware stack: SM120 refers to the compute capability of NVIDIA Blackwell GPUs (RTX PRO 6000). SM100 is Hopper (H100/H200). These have different instruction sets and require different CUDA code generation.
  2. The software stack: SGLang is an inference engine for LLMs. Its sgl-kernel package contains CUDA-optimized kernels (attention, MoE, allreduce, etc.) that must be compiled for the target GPU architecture.
  3. The build system: The project uses CMake with scikit-build-core. Environment variables like SGL_KERNEL_CUDA_ARCHS are meant to control which GPU architectures to target, but the CMakeLists.txt may or may not respect them.
  4. The session history: The assistant had already tried an editable install (which produced no .so files), then a non-editable install (which produced .so files but only for SM90 and SM100). The timestamp "22:32" confirms the binary was just rebuilt.
  5. The load_utils.py architecture: Earlier in the session, the assistant discovered that load_utils.py maps SM120 to the sm100 directory — meaning the runtime expects SM120-compatible code to live in the sm100/ subdirectory, not a separate sm120/ directory.

Assumptions and Their Consequences

The message reveals several assumptions, some correct and some not:

Correct assumption: The CMakeLists.txt is the authoritative source for what architectures get compiled. Checking it directly is the right approach.

Incorrect assumption: The SGL_KERNEL_CUDA_ARCHS environment variable controls which architectures CMake targets. The grep output shows that the CMakeLists.txt hard-codes both SM100 and SM120 flags, suggesting the env var is either not read by this version of the build system or is overridden by the hard-coded values.

Implicit assumption: That the build produced a binary without SM120 support. In reality, the binary likely does contain SM120 code (since the CMake flags include compute_120a,code=sm_120a), but the runtime loader can't find it because it's looking in the wrong directory or because of the undefined symbol error encountered earlier.

Unstated assumption: That the undefined symbol error from the previous message (msg 3123) was caused by missing SM120 code. In fact, that error was libtorch.so: cannot open shared object file: No such file or directory — a library path issue, not an architecture mismatch.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is a masterclass in systematic debugging. The sequence of thought is:

  1. Observation: The binary was rebuilt but still doesn't work on SM120.
  2. Hypothesis: The SGL_KERNEL_CUDA_ARCHS env var was ignored.
  3. Investigation: Check the CMakeLists.txt to see how architectures are configured.
  4. Analysis: The CMake file contains both SM100 and SM120 flags — the build system already supports SM120.
  5. Implicit conclusion: The problem is not the build but the runtime loading. This reasoning chain is visible in the transition from the opening statement ("It was rebuilt... But CMake built it for SM100, not SM120") to the grep command. The assistant doesn't just accept the failure — it traces the cause upstream to the build configuration.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. The CMakeLists.txt already supports SM120: The -gencode=arch=compute_120a,code=sm_120a flag confirms that Blackwell compilation is built into the source. No patches or modifications are needed to add SM120 support.
  2. The env var is not authoritative: SGL_KERNEL_CUDA_ARCHS="120" does not restrict the build to SM120 alone. The CMake file hard-codes both architectures.
  3. The binary contains both SM100 and SM120 code: Since both flags are passed to nvcc, the resulting .so file is a fat binary containing GPU code for both architectures.
  4. The real problem is elsewhere: The grep results redirect attention from the build system to the runtime loader. If the binary already contains SM120 code, the issue must be in how load_utils.py discovers and loads the correct variant.

The Pivot That Follows

In the very next message (msg 3126), the assistant acts on this new understanding. It realizes: "The CMake already builds for SM120 — compute_120a,code=sm_120a. The problem is the load_utils.py maps SM120 → sm100 directory, and the built binary goes into sm100/. So the binary should actually contain SM120 code! The issue was the undefined symbol error."

The assistant then tests loading sgl_kernel after importing torch first — and it works. The entire SM120 "build failure" was a red herring. The binary was fine all along; the real issue was a missing libtorch.so in the library path when sgl_kernel was imported standalone.

Broader Significance

Message 3125 exemplifies a pattern that recurs throughout systems engineering: the most valuable debugging moments are not the ones that fix the problem, but the ones that reframe it. The assistant spent 48 minutes building a binary it didn't need to build, because the real problem was a library path issue that would have taken seconds to diagnose. Yet the build effort was not wasted — it forced the assistant to examine the build system, discover the SM120 support already present, and ultimately trace the issue to its true source.

This message also highlights the fragility of the ML infrastructure stack. A single missing shared library (libtorch.so) can masquerade as a GPU architecture incompatibility, wasting hours of engineering time. The assistant's willingness to question its own assumptions — to check whether the env var was actually respected — is what ultimately broke the logjam.

For anyone working with large language model deployment on novel hardware, this message offers a cautionary tale: when a build "fails," first verify that it actually failed. The binary might be perfectly compiled but simply unable to load due to a missing dependency. And when an environment variable doesn't work as expected, trace it through the build system — the answer is often in the CMakeLists.txt.