The Blackwell Kernel Puzzle: Debugging SM120 Compatibility in SGLang's sgl-kernel

In the high-stakes world of deploying 1-trillion-parameter language models, every optimization matters — and every hardware compatibility issue becomes a potential showstopper. Message 3123 captures a pivotal debugging moment in an opencode coding session where an AI assistant is wrestling with the bleeding edge of GPU compute: getting SGLang's sgl-kernel to work on NVIDIA's Blackwell (SM120) architecture. This single message, a mix of analysis and experimental probing, reveals the intricate dance between software assumptions and hardware reality when deploying cutting-edge models on cutting-edge hardware.

The Strategic Context: Why This Message Matters

To understand why this message was written, we must first understand the larger mission. The session had been building toward speculative decoding for Kimi-K2.5, a massive 1-trillion-parameter Mixture-of-Experts model deployed across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The team had invested heavily in the EAGLE-3 speculative decoding pipeline — generating 10,000 synthetic training samples, extracting hidden states totaling 828 GB of data, and fine-tuning a draft model over five epochs. But when they integrated the trained drafter with vLLM, they hit a wall: only a 15% token acceptance rate, yielding 0.66x throughput — worse than running without speculation at all.

The user directed a pivot to SGLang, which promised first-class EAGLE-3 support with reported 1.8x speedups on Kimi-K2 architectures. But SGLang had never been tested on Blackwell (SM120) GPUs. The assistant had just spent 48 minutes building sgl-kernel from source, only to find that the resulting binary wouldn't load. Message 3123 is the moment the assistant stops throwing build commands at the problem and starts thinking about what's actually going wrong.

The Detective Work: Tracing Through the Code

The message opens with the assistant articulating its newfound understanding of the problem:

I see — for SM120 (compute_capability=120), it sets ops_subdir = "sm100" (the else branch catches everything ≥100 that's not 90). But the SM100 binary fails on SM120 due to an undefined symbol. The SM100 binary was compiled for SM100 only, not SM120.

This insight came from reading the load_utils.py code in the previous message ([msg 3122]), where the assistant examined lines 50-110 of the architecture detection logic. The code uses a simple conditional: if compute capability equals 90, load from sm90/; otherwise (for anything ≥100), load from sm100/. This design assumes that SM100 binaries are forward-compatible with future architectures — an assumption that was baked into the code before Blackwell existed.

The assistant correctly identifies the immediate symptom: the SM100 binary fails on SM120 with an undefined symbol error. But the reasoning about why reveals an interesting tension. The assistant assumes "the SM100 binary was compiled for SM100 only, not SM120" — a reasonable inference given the error. However, as later messages reveal ([msg 3125], [msg 3126]), the CMake build system does include SM120 code generation targets (compute_120a,code=sm_120a). The real problem is more nuanced: the build system puts the binary in the sm100/ directory regardless of which architectures it was compiled for, and the loading code maps SM120 to that same directory. The undefined symbol error was either a red herring or a consequence of how the binary was loaded without the proper library path.

The Experimental Probe: Testing Forward Compatibility

The assistant then formulates a testable hypothesis: "Let me check if the sm100 binary can be loaded by tweaking things, or if I can just symlink/copy and it'll work since SM120 is forward-compatible from SM100."

This is a critical moment of reasoning. The assistant considers two possible solutions:

  1. Make the existing SM100 binary work on SM120 — perhaps by patching the loading logic or adjusting library paths
  2. Build a separate SM120 binary — the more thorough but time-consuming approach The forward-compatibility assumption is worth examining. In NVIDIA's GPU architecture roadmap, each compute capability version typically includes all instructions from previous versions. SM120 (Blackwell) should be able to execute SM100 (Hopper) code. However, the undefined symbol error suggests something more specific is failing — perhaps a device-specific intrinsic or a library dependency that differs between architectures. The assistant's experimental approach is methodical: directly import the .so file using Python's importlib machinery, bypassing the sgl_kernel package's architecture detection logic entirely. This isolates the problem: if the binary loads successfully via direct import, the issue is in the loading/fallback logic; if it still fails, the binary itself is incompatible.
import importlib.util, pathlib
ops_path = pathlib.Path("/root/ml-env/lib/python3.12/site-packages/sgl_kernel/sm100/common_ops.abi3.so")
spec = importlib.util.spec_from_file_location("common_ops", str(ops_path))
common_ops = importlib.util.module_from_spec(spec)
spec.loader.exec_module(common_ops)
print("Loaded OK")

The Unexpected Error: A Red Herring or a Clue?

The experiment yields a different error than expected:

ImportError: libtorch.so: cannot open shared object file: No such file or directory

This is a library path issue, not an architecture incompatibility. The .so file depends on libtorch.so, which isn't in the standard library search path when Python loads the binary directly. The sgl_kernel package normally handles this by importing torch first (which loads libtorch.so into the process), but the direct import bypasses that initialization.

This result is both a setback and a clue. It tells the assistant that:

  1. The binary can be loaded — the architecture compatibility issue might not be the primary problem
  2. The loading mechanism matters — the sgl_kernel package's initialization sequence is important
  3. The earlier "undefined symbol" error might have been a consequence of how the binary was loaded, not what it contained In the following messages ([msg 3124], [msg 3125], [msg 3126]), the assistant builds on this insight. It checks whether the build actually produced new binaries (it did — the timestamp was updated), discovers that CMake does compile for SM120, and finally succeeds by importing torch before sgl_kernel. The resolution is almost anticlimactic: sgl_kernel loaded OK! — the binary was fine all along, the issue was just the loading sequence.

Assumptions, Mistakes, and Lessons

This message reveals several assumptions that shaped the debugging process:

The undefined symbol assumption. The assistant assumed the SM100 binary was compiled exclusively for SM100 and lacked SM120 support. This was wrong — the CMake build system included SM120 code generation targets. The error message was misleading, pointing to an architecture issue when the real problem was a library path issue.

The forward-compatibility assumption. The assistant assumed SM120 is forward-compatible from SM100, meaning SM100-compiled code should run on SM120. This is generally true for NVIDIA GPUs (each generation adds instructions but doesn't remove them), but the assumption needed verification.

The build system assumption. The assistant assumed that setting SGL_KERNEL_CUDA_ARCHS="120" would control which architectures the binary was compiled for. In reality, the environment variable wasn't being respected by the CMake build — the binary was always being compiled for both SM100 and SM120 regardless.

The mistake here is subtle but instructive: the assistant conflated the loading problem (undefined symbol when loading on SM120) with a build problem (binary not compiled for SM120). The actual issue was that the binary was compiled for SM120, but the loading code had a library path dependency that wasn't being satisfied. The direct import experiment inadvertently revealed this, though the assistant didn't fully recognize it until the next round of messages.

Input Knowledge Required

To fully understand this message, one needs:

  1. CUDA compute capability architecture: Understanding that SM90 = Hopper (H100), SM100 = Hopper Next, SM120 = Blackwell. Each generation has different instruction sets and capabilities.
  2. SGLang's kernel loading architecture: The sgl-kernel package uses architecture-specific shared libraries in subdirectories (sm90/, sm100/), with a Python loader that detects the GPU and selects the appropriate binary.
  3. Python importlib mechanics: The assistant uses importlib.util.spec_from_file_location and module_from_spec to directly load a shared object, bypassing the package's normal initialization.
  4. ELF shared library dependencies: The libtorch.so error reveals that .so files have runtime dependencies that must be satisfied, typically by having the dependent library already loaded in the process.
  5. The broader deployment context: The Kimi-K2.5 model, EAGLE-3 speculative decoding, the pivot from vLLM to SGLang, and the Blackwell GPU hardware.

Output Knowledge Created

This message produces several pieces of knowledge:

  1. The architecture mapping is wrong for SM120: The load_utils.py maps SM120 to the sm100/ directory, which might or might not work depending on how the binary was compiled.
  2. The SM100 binary can be loaded directly: The direct import attempt shows the binary is loadable (given proper library paths), suggesting the architecture issue might not be fundamental.
  3. Library path dependencies matter: The libtorch.so error reveals that sgl-kernel's loading depends on torch being initialized first — a dependency that the normal import sequence handles but direct loading bypasses.
  4. A debugging methodology for kernel loading: The assistant demonstrates a technique for isolating architecture compatibility issues from library path issues by using direct importlib loading.

The Thinking Process Visible in the Reasoning

What makes this message particularly interesting is the visible arc of the assistant's reasoning. It moves from:

  1. Observation: The SM100 binary fails on SM120 with an undefined symbol error.
  2. Analysis: The load_utils.py maps SM120 to sm100/ via an else branch.
  3. Hypothesis: The SM100 binary was compiled only for SM100, not SM120.
  4. Prediction: SM120 might be forward-compatible from SM100, so a symlink/copy approach might work.
  5. Experiment: Direct import of the SM100 binary to test loadability.
  6. Surprise: A different error (libtorch.so) appears, not the undefined symbol.
  7. Implication: The loading mechanism itself is part of the problem. This is classic debugging: each experiment refines the hypothesis, and unexpected results are often more informative than expected ones. The assistant doesn't jump to conclusions — it lets the code speak, even when the answers are confusing.

Conclusion

Message 3123 is a snapshot of a debugging session at the frontier of AI infrastructure. The assistant is trying to deploy a state-of-the-art speculative decoding system on hardware that's so new that the software ecosystem hasn't fully caught up. The sgl-kernel's architecture detection code was written before Blackwell existed, and its fallback logic — while reasonable at the time — creates a compatibility trap for early adopters of SM120 hardware.

The message demonstrates that debugging at this level requires not just knowledge of the codebase but an understanding of the entire stack: GPU architecture, build systems, shared library loading, and Python's import machinery. It also shows how assumptions can lead us astray — the "undefined symbol" error pointed toward an architecture incompatibility, but the real issue was a library path problem that was resolved simply by importing torch first.

In the end, the assistant's methodical approach paid off. The sgl-kernel loaded successfully in the next round of messages ([msg 3127]), and SGLang was able to proceed with deploying the Kimi-K2.5 model. But the journey through this single message reveals the complexity of working at the bleeding edge, where every layer of abstraction can become a debugging target, and where the difference between success and failure often comes down to understanding a single line of fallback logic in a loader script.