When Bleeding Edge Meets Bleeding Edge: Building sgl-kernel for SM120 on Blackwell GPUs

"Compatibility with CMake < 3.5 has been removed from CMake."

This innocuous-looking error message, buried in the tail of a 48-minute build attempt, encapsulates a fundamental tension in modern AI infrastructure: what happens when your hardware is so new that the build tools themselves haven't caught up? Message 3109 of this opencode session captures a single moment in a much longer struggle — the attempt to build sgl-kernel for NVIDIA's SM120 architecture (compute capability 12.0, found on RTX PRO 6000 Blackwell GPUs) — and the failure that ensued is a masterclass in the hidden complexities of deploying bleeding-edge machine learning systems.

The Message

The subject message is a single bash command, issued by the AI assistant to a remote server at 10.1.230.174:

ssh root@10.1.230.174 'CUDA_HOME=/usr/local/cuda-12.8 CUDACXX=/usr/local/cuda-12.8/bin/nvcc PATH=/usr/local/cuda-12.8/bin:$PATH SGL_KERNEL_CUDA_ARCHS="120" ~/.local/bin/uv pip install --python ~/ml-env/bin/python3 -e /root/sglang/sgl-kernel/ --no-build-isolation 2>&1 | tail -20'

The output shows a CMake error from the dlpack subproject:

CMake Error at /tmp/tmpu1q0e7tj/build/_deps/dlpack-src/CMakeLists.txt:5
(cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max>
  syntax to tell CMake that the project requires at least <min> but has been
  updated to work with policies introduced by <max> or earlier.

On its surface, this is a simple CMake version incompatibility. But to understand why this message was written — and why it matters — we need to trace the full chain of reasoning that led to this moment.

The Context: A Long Road to Speculative Decoding

This message is not an isolated incident. It sits at the end of a long and arduous journey spanning multiple sessions of work on an 8× Blackwell GPU server. The team had been attempting to deploy speculative decoding (specifically EAGLE-3) for Kimi-K2.5, a massive 1-trillion-parameter Mixture-of-Experts language model. The goal was straightforward: use a smaller "draft" model to predict the larger model's outputs, achieving 1.5–2× throughput improvement without sacrificing quality.

The path had been anything but straightforward. The team had:

  1. Built a complete EAGLE-3 training pipeline — generating 10,000 synthetic training samples, extracting hidden states (producing 828 GB of data), and fine-tuning a drafter model for 5 epochs.
  2. Tested it with vLLM — only to discover a devastating ~15% acceptance rate, yielding 0.66× throughput (slower than running without speculation at all).
  3. Confirmed the problem was not their training — the pre-trained AQ-MedAI baseline drafter performed identically poorly, pointing to a fundamental vLLM integration bug with DeepSeek V3's Multi-head Latent Attention (MLA) architecture.
  4. Pivoted to SGLang — which has first-class EAGLE-3 support, is explicitly tested with Kimi-K2 drafters, and reportedly achieves 1.8× speedup. The pivot to SGLang was the user's explicit directive in message 3093: "Get the models up on SGLang, if acceptance rate seems low try to debug things like data and predictions." This message, 3109, is the third attempt to build the SM120-compatible sgl-kernel that SGLang requires.

Why This Specific Build Failed

The failure is a classic dependency version mismatch. The sgl-kernel project pulls in dlpack as a dependency via CMake's FetchContent. The dlpack project's CMakeLists.txt uses cmake_minimum_required(VERSION 2.0), which specifies compatibility with CMake versions as old as 2.0. However, CMake 4.2.1 (installed on the system) has removed support for CMake versions earlier than 3.5 — a breaking change introduced in the CMake 4.x series.

This is a particularly subtle failure mode because:

Assumptions and Their Consequences

Several assumptions underpinned this build attempt, and several of them proved incorrect:

Assumption 1: The build would work out of the box. The assistant had verified that sgl-kernel's CMakeLists.txt explicitly includes SM120 support (-gencode=arch=compute_120a,code=sm_120a), and that CUDA 12.8 supports SM120. The assumption that a clean build would succeed was reasonable but failed to account for transitive dependency issues.

Assumption 2: The -e (editable) install would produce compiled binaries. Earlier attempts (messages 3103–3107) had failed for different reasons — missing scikit-build-core, missing libnuma-dev, and then this CMake error. The assistant was iterating through build failures, each revealing a new missing dependency or incompatibility.

Assumption 3: The error was in the main build, not a subproject. The tail -20 output only shows the last 20 lines of the build log. The actual error from dlpack appears there, but the full context — that this is a fetched dependency, not part of sgl-kernel itself — is not immediately obvious. A reader unfamiliar with CMake's FetchContent might think the error is in sgl-kernel's own build configuration.

Assumption 4: SM120 support in the source means SM120 support in the build. This turned out to be only partially true. Even after the CMake issue was resolved (in subsequent messages), the build produced only SM90 and SM100 binaries — no SM120 variant. The build system's architecture detection and fallback logic had its own issues.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Knowledge of the broader project: That this is part of deploying EAGLE-3 speculative decoding for Kimi-K2.5 on 8× Blackwell GPUs, and that the team has already ruled out vLLM due to a fundamental integration bug.
  2. Understanding of SM120/Blackwell: That SM120 (compute capability 12.0) is NVIDIA's newest architecture, found in RTX PRO 6000 Blackwell GPUs, and that software support for it is still maturing. The SGL_KERNEL_CUDA_ARCHS=&#34;120&#34; environment variable tells the build system to target only SM120, skipping other architectures to save compilation time.
  3. Familiarity with the build toolchain: The command uses uv pip install (a fast Python package manager), --no-build-isolation (to use the existing environment rather than creating a temporary isolated build environment), and -e (editable install, which links the source directory into the Python environment rather than copying built artifacts). The environment variables CUDA_HOME, CUDACXX, and PATH are all set to ensure the correct CUDA 12.8 toolkit is used.
  4. Awareness of CMake's version compatibility model: The cmake_minimum_required(VERSION X.Y) directive in a CMakeLists.txt tells CMake what version policies to enable. CMake 4.x removed support for policies from versions earlier than 3.5, which is what triggers this specific error.
  5. Context about the machine: That this is a remote server with 8× RTX PRO 6000 Blackwell GPUs, running Ubuntu 24.04 with CUDA 12.8 and PyTorch 2.10.0, and that earlier build attempts had OOM'd the container (message 3111: "oom-ing pretty bad, try -j20").

Output Knowledge Created

This message creates several pieces of knowledge:

  1. The build fails with a specific CMake error: The error message itself is diagnostic output that will inform the next debugging step. The assistant now knows that dlpack's CMakeLists.txt is incompatible with CMake 4.x.
  2. A workaround path is suggested: The error message helpfully suggests updating the VERSION argument or using the &lt;min&gt;...&lt;max&gt; syntax. This points toward setting CMAKE_POLICY_VERSION_MINIMUM=3.5 as an environment variable, which the assistant attempts in the very next message (3110).
  3. Confirmation that the build infrastructure is functional: Despite the failure, the build got past several earlier hurdles — it found CUDA 12.8, it configured the main project, it fetched dependencies. The failure is isolated to a specific compatibility issue rather than a systemic problem.
  4. Evidence of the bleeding-edge nature of the setup: The fact that CMake 4.2.1 (the latest) is being used with a project that specifies compatibility with CMake 2.0 (from 2012) illustrates the wide gap between the cutting-edge hardware (SM120) and the legacy build configurations of upstream dependencies.

The Thinking Process

The reasoning visible in this message and its surrounding context reveals a methodical, iterative debugging approach:

Step 1: Identify the target. The assistant knows it needs sgl-kernel compiled for SM120. The installed version (from message 3098) only supports SM100, and importing it fails with ImportError.

Step 2: Verify source support. The assistant checks the CMakeLists.txt (message 3102) and confirms SM120 is explicitly listed as a target architecture. Good — the source supports it.

Step 3: Attempt build. The first build attempt (message 3103) fails because scikit-build-core is missing. The assistant installs it (3104) and tries again.

Step 4: Iterate through failures. The second attempt (3105) fails with a NUMA library error. The assistant installs libnuma-dev (3108) and tries again.

Step 5: The subject message. The third attempt (3109) fails with the CMake/dlpack error. The assistant has now identified a new class of problem — not a missing system library, but a CMake version incompatibility in a fetched dependency.

Step 6: Apply workaround. In the next message (3110), the assistant tries setting CMAKE_POLICY_VERSION_MINIMUM=3.5 as an environment variable, which is the standard CMake workaround for this exact issue.

What's notable about this process is the assistant's patience. Each build attempt takes 48 minutes (as revealed in message 3115: "Prepared 1 package in 48m 32s"). The assistant is committing to hour-long build cycles, iterating through failures one at a time. This is not a debugging process that could be done interactively — it requires planning, patience, and the ability to diagnose errors from log output alone.

Mistakes and Incorrect Assumptions

Several aspects of this approach could be questioned:

The tail -20 truncation. The assistant uses tail -20 to limit output, which is sensible for a long build log but risks missing important context. In this case, the error message is captured, but the full chain of CMake's FetchContent download and configuration steps is hidden. A more verbose output might have revealed the dlpack dependency earlier.

The editable install (-e). The assistant uses -e for an editable install, which is unusual for a CUDA-heavy package. Editable installs work by linking the source directory into site-packages, but compiled .so files need to be built into a specific location. As later messages reveal (3117–3120), the editable install didn't produce any .so files at all — the build artifacts were missing. A non-editable install (pip install /root/sglang/sgl-kernel/ without -e) eventually worked but still didn't produce SM120 binaries.

Assuming the build system handles SM120 correctly. The sgl-kernel build system has architecture-specific logic that maps SM architectures to subdirectories (sm90/, sm100/). The assistant assumed that setting SGL_KERNEL_CUDA_ARCHS=&#34;120&#34; would produce an sm120/ directory, but the build system's architecture detection and file organization didn't account for SM120 as a separate variant. This required further debugging in subsequent messages.

Not checking CMake version compatibility upfront. A more experienced CMake user might have anticipated the dlpack compatibility issue, given that CMake 4.x is known to break backward compatibility. Setting CMAKE_POLICY_VERSION_MINIMUM=3.5 preemptively could have saved a build cycle.

The Broader Significance

This message is a microcosm of the challenges faced by anyone deploying AI models on cutting-edge hardware. The Blackwell GPU (SM120) is so new that:

Conclusion

Message 3109 is a snapshot of a moment in time — 48 minutes of compilation, distilled into 20 lines of error output. It represents the third iteration of a build process that would ultimately require multiple more attempts before sgl-kernel was successfully compiled for SM120. The CMake error itself is trivial to fix (set CMAKE_POLICY_VERSION_MINIMUM=3.5), but discovering which fix is needed requires understanding the full dependency chain, reading error messages carefully, and having the patience to iterate through hour-long build cycles.

In the end, the build did succeed (message 3115: "Built sgl-kernel @ file:///root/sglang/sgl-kernel" in 48m 32s), but it produced only SM90 and SM100 binaries — no SM120 variant. The journey to get EAGLE-3 working on Blackwell GPUs was far from over. But this message, with its deceptively simple CMake error, captures the essence of what it means to work at the bleeding edge: every layer of the stack is a potential failure point, and progress comes from methodically eliminating each one.