The Missing SM120: A 48-Minute Build Reveals a Silent Failure

The Message

ssh root@10.1.230.174 'find /root/ml-env/lib/python3.12/site-packages/sgl_kernel/ -name "*.so" 2>/dev/null | head -10; echo "---"; ls /root/ml-env/lib/python3.12/site-packages/sgl_kernel/sm*/ 2>/dev/null | head -10'

Output:

/root/ml-env/lib/python3.12/site-packages/sgl_kernel/flashmla_ops.abi3.so
/root/ml-env/lib/python3.12/site-packages/sgl_kernel/sm100/common_ops.abi3.so
/root/ml-env/lib/python3.12/site-packages/sgl_kernel/sm90/common_ops.abi3.so
/root/ml-env/lib/python3.12/site-packages/sgl_kernel/flash_ops.abi3.so
/root/ml-env/lib/python3.12/site-packages/sgl_kernel/spatial_ops.abi3.so
---
/root/ml-env/lib/python3.12/site-packages/sgl_kernel/sm100/:
common_ops.abi3.so

/root/ml-env/lib/python3.12/site-packages/...

This message, appearing at index 3120 in the conversation, is a deceptively simple diagnostic command that reveals a deeply frustrating outcome: after nearly an hour of compilation, the SM120 (Blackwell GPU) kernel variant that the assistant explicitly requested has silently failed to materialize.

Context: The Pivot to SGLang

To understand why this message exists, we must trace the events that led to it. The session had been pursuing speculative decoding for the Kimi-K2.5 model — a 1-trillion-parameter Mixture-of-Experts language model running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 compute capability). The assistant had invested enormous effort building a complete EAGLE-3 training pipeline: generating 10,000 synthetic training samples over 5.3 hours, extracting hidden states at 3,165 tok/s producing 828 GB of training data, and fine-tuning a draft model for 5 epochs in 2.6 hours.

The payoff was devastating. When tested with vLLM's EAGLE-3 integration, both the newly trained drafter and the pre-trained AQ-MedAI baseline achieved only ~15% token acceptance rate — resulting in 0.66x throughput, meaning speculative decoding made the model slower than running without it. The root cause was identified as a fundamental integration issue between vLLM's EAGLE-3 implementation and DeepSeek V3's Multi-head Latent Attention (MLA) architecture: the auxiliary hidden states being extracted during decode were misaligned or incorrect.

The user directed the assistant to pivot to SGLang ([msg 3093]), which has first-class EAGLE-3 support explicitly tested with Kimi-K2 drafters and reported to achieve 1.8x speedup. This pivot set in motion a new chain of work: stop the vLLM service, install SGLang, build the required sgl-kernel for SM120, and test the model.

The Build That Wasn't

The immediate predecessor to this diagnostic message was a 48-minute sgl-kernel build ([msg 3119]). The assistant had already attempted an editable install ([msg 3115]) that appeared to succeed but produced no compiled .so files — the editable install placed the build artifacts in the source tree, not in site-packages. A subsequent non-editable install with --reinstall ([msg 3119]) also completed successfully, with the build system printing "Built sgl-kernel" and the package manager reporting it as installed.

Both builds were invoked with the environment variable SGL_KERNEL_CUDA_ARCHS=&#34;120&#34;, explicitly requesting compilation for SM120 (Blackwell) architecture. The build system accepted this input without complaint. The CMAKE_BUILD_PARALLEL_LEVEL=20 and MAX_JOBS=20 flags were set to prevent the out-of-memory crashes that had plagued earlier compilation attempts (<msg id=3111-3113>). Everything appeared to go smoothly.

But appearances were deceiving.

The Diagnostic

Message 3120 is the moment of truth. The assistant runs a simple find command to enumerate the compiled shared objects in the installed package directory, followed by an ls of the architecture-specific subdirectories. The command is structured to show two things: first, all .so files regardless of location, and second, the contents of any sm* subdirectories (which would be sm90, sm100, and ideally sm120).

The output confirms the presence of sm90/common_ops.abi3.so and sm100/common_ops.abi3.so — the compiled kernels for Hopper (SM90) and Blackwell (SM100) architectures. But there is no sm120/ directory. No sm120/common_ops.abi3.so. The SM120 variant simply does not exist.

This is a classic "silent failure" in build systems: the build completed successfully from the build system's perspective, but it did not produce the requested output. The SGL_KERNEL_CUDA_ARCHS=&#34;120&#34; environment variable was either ignored, overridden, or processed in a way that caused the build to fall back to default architectures without raising an error.

Assumptions and Their Failure

Several assumptions underpin this diagnostic message, and several of them turned out to be incorrect:

Assumption 1: The build system respects SGL_KERNEL_CUDA_ARCHS. The assistant assumed that setting this environment variable would cause the CMake build system to target SM120. However, the build system may have had a hardcoded list of architectures, or the variable may have been read differently than expected. The CMakeLists.txt did contain SM120 references ([msg 3102]), but the mechanism for selecting architectures at build time may not have been properly wired.

Assumption 2: A successful build message means the build produced the expected artifacts. The build system printed "Built sgl-kernel" and the package manager reported installation success. Neither the assistant nor the build system detected that the SM120 variant was missing. This is a systemic issue: build systems typically report success based on compilation exit codes, not on whether specific expected outputs exist.

Assumption 3: The non-editable install would place artifacts in site-packages. The editable install ([msg 3115]) had placed compiled files in the source tree rather than site-packages, which was why the initial verification ([msg 3116]) failed. The assistant correctly identified this and switched to a non-editable install ([msg 3119]). This did place files in site-packages — but still without SM120 support.

Assumption 4: SM120 support exists in the sgl-kernel source. This was verified earlier ([msg 3102]) by finding SM120 references in CMakeLists.txt. The source does have SM120 support — the issue is that the build configuration didn't activate it.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. The concept of CUDA compute capabilities (SM architectures). SM90 = Hopper (H100), SM100 = Blackwell (B200/RTX PRO 6000), SM120 = next-generation Blackwell variant. The RTX PRO 6000 Blackwell GPUs in this system report SM120, which is bleeding-edge hardware.
  2. The sgl-kernel architecture. SGLang's kernel library uses architecture-specific subdirectories (sm90/, sm100/, etc.) containing compiled shared objects. The load_utils.py module dynamically loads the appropriate variant based on the GPU's compute capability. Without an SM120 variant, the library cannot load on SM120 hardware.
  3. The history of build failures. Earlier attempts to build sgl-kernel failed due to missing dependencies (scikit-build-core, libnuma-dev) and CMake version incompatibilities (<msg id=3103-3109>). The container also OOM'd during compilation (<msg id=3111-3113>), requiring the user to force-restart it. Each of these failures was resolved, but the final build still didn't produce the right output.
  4. The broader context of the EAGLE-3 speculative decoding effort. This build is part of a pivot from vLLM (which failed to achieve useful speculation) to SGLang (which promises 1.8x speedup). The SM120 build is a prerequisite for even testing whether SGLang works on this hardware.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. The sgl-kernel build system silently ignores SGL_KERNEL_CUDA_ARCHS=&#34;120&#34;. This is the primary finding. The environment variable that should control architecture targeting is not being honored.
  2. The installed package contains SM90 and SM100 variants but not SM120. The existing sm100/ directory contains common_ops.abi3.so for the earlier Blackwell generation (SM100), but the SM120 variant needed for the RTX PRO 6000 Blackwell GPUs is absent.
  3. The build system has a gap in its architecture selection logic. Whether this is a CMake configuration issue, a Python build script problem, or a missing mapping in the architecture detection code, there is a disconnect between the user's intent and the build output.
  4. The verification approach needs to change. The assistant had been relying on import tests (import sgl_kernel) to verify the build, but this fails with a generic "Could not load any common_ops library" error that doesn't pinpoint the architecture mismatch. The find command in this message provides much more specific diagnostic information.

The Thinking Process

The assistant's reasoning in this message is a model of systematic debugging. After the editable install produced no .so files at all (<msg id=3117-3118>), the assistant correctly deduced that editable installs place artifacts in the source tree rather than site-packages. The switch to a non-editable install with --reinstall was the logical next step.

When that build also appeared to succeed, the assistant didn't simply trust the build output — it performed a targeted verification. The choice of commands is revealing:

The Broader Significance

This message captures a universal experience in ML engineering: the silent build failure. A 48-minute compilation completes without errors, the package manager reports success, but the critical architecture-specific binary is missing. No warning, no error, no indication that anything went wrong — until the runtime import fails on the target hardware.

The debugging approach here is instructive. Rather than re-running the build with different flags (which would waste another 48 minutes), the assistant performs a targeted inspection of the build output. The find command is fast, precise, and reveals exactly what's missing. This is the difference between debugging by trial and error and debugging by investigation.

For the broader EAGLE-3 effort, this message marks a critical juncture. The pivot to SGLang was motivated by vLLM's failure to handle MLA attention correctly for speculative decoding. But SGLang cannot even be tested until the kernel library supports the target hardware. The SM120 build issue becomes a gate — until it's resolved, the entire speculative decoding effort is blocked on this one missing .so file.

The message also highlights the challenges of working with bleeding-edge hardware. SM120 is so new that even the build tools designed to support it have gaps. The SGL_KERNEL_CUDA_ARCHS environment variable exists precisely to handle new architectures, but its implementation is incomplete. This is the cost of being on the frontier: the tools are not yet fully baked, and every build is an adventure.